2015-08-18 17:07:38,310 INFO [main] util.VersionInfo: HBase 1.1.1 2015-08-18 17:07:38,311 INFO [main] util.VersionInfo: Source code repository git://hw11397.local/Volumes/hbase-1.1.1RC0/hbase revision=d0a115a7267f54e01c72c603ec53e91ec418292f 2015-08-18 17:07:38,311 INFO [main] util.VersionInfo: Compiled by ndimiduk on Tue Jun 23 14:44:07 PDT 2015 2015-08-18 17:07:38,311 INFO [main] util.VersionInfo: From source with checksum 6e2d8cecbd28738ad86daacb25dc467e 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:HBASE_CLASSPATH=/home/aeris/hadoop-2.7.1/etc/hadoop 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:HBASE_HEAPSIZE=10000 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:TERM=xterm 2015-08-18 17:07:38,598 INFO [main] util.ServerCommandLine: env:USER=aeris 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:CLASSPATH=/home/aeris/hbase-1.1.1/bin/../conf:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/tools.jar:/home/aeris/hbase-1.1.1/bin/..:/home/aeris/hbase-1.1.1/bin/../lib/activation-1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/aopalliance-1.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/aeris/hbase-1.1.1/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/aeris/hbase-1.1.1/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/aeris/hbase-1.1.1/bin/../lib/api-util-1.0.0-M20.jar:/home/aeris/hbase-1.1.1/bin/../lib/asm-3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/avro-1.7.4.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-beanutils-1.7.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-cli-1.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-codec-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-collections-3.2.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-compress-1.4.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-configuration-1.6.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-daemon-1.0.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-digester-1.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-el-1.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-httpclient-3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-io-2.4.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-lang-2.6.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-logging-1.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-math-2.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-math3-3.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-net-3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/disruptor-3.3.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/aeris/hbase-1.1.1/bin/../lib/guava-12.0.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/guice-3.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/guice-servlet-3.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-annotations-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-auth-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-client-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-hdfs-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-api-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-client-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-annotations-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-annotations-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-client-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-common-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-common-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-examples-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-hadoop-compat-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-hadoop2-compat-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-it-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-it-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-prefix-tree-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-procedure-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-protocol-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-rest-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-server-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-server-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-shell-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-thrift-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/aeris/hbase-1.1.1/bin/../lib/httpclient-4.2.5.jar:/home/aeris/hbase-1.1.1/bin/../lib/httpcore-4.1.3.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-core-asl-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-xc-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jamon-runtime-2.3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/jasper-compiler-5.5.23.jar:/home/aeris/hbase-1.1.1/bin/../lib/jasper-runtime-5.5.23.jar:/home/aeris/hbase-1.1.1/bin/../lib/java-xmlbuilder-0.4.jar:/home/aeris/hbase-1.1.1/bin/../lib/javax.inject-1.jar:/home/aeris/hbase-1.1.1/bin/../lib/jaxb-api-2.2.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/aeris/hbase-1.1.1/bin/../lib/jcodings-1.0.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-client-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-core-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-guice-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-json-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-server-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jets3t-0.9.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/jettison-1.3.3.jar:/home/aeris/hbase-1.1.1/bin/../lib/jetty-6.1.26.jar:/home/aeris/hbase-1.1.1/bin/../lib/jetty-sslengine-6.1.26.jar:/home/aeris/hbase-1.1.1/bin/../lib/jetty-util-6.1.26.jar:/home/aeris/hbase-1.1.1/bin/../lib/joni-2.1.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/jruby-complete-1.6.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsch-0.1.42.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsp-2.1-6.1.14.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsr305-1.3.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/junit-4.11.jar:/home/aeris/hbase-1.1.1/bin/../lib/leveldbjni-all-1.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/libthrift-0.9.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/log4j-1.2.17.jar:/home/aeris/hbase-1.1.1/bin/../lib/metrics-core-2.2.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/netty-3.2.4.Final.jar:/home/aeris/hbase-1.1.1/bin/../lib/netty-all-4.0.23.Final.jar:/home/aeris/hbase-1.1.1/bin/../lib/paranamer-2.3.jar:/home/aeris/hbase-1.1.1/bin/../lib/protobuf-java-2.5.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/aeris/hbase-1.1.1/bin/../lib/servlet-api-2.5.jar:/home/aeris/hbase-1.1.1/bin/../lib/slf4j-api-1.7.7.jar:/home/aeris/hbase-1.1.1/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/aeris/hbase-1.1.1/bin/../lib/snappy-java-1.0.4.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/spymemcached-2.11.6.jar:/home/aeris/hbase-1.1.1/bin/../lib/xmlenc-0.52.jar:/home/aeris/hbase-1.1.1/bin/../lib/xz-1.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/zookeeper-3.4.6.jar::/home/aeris/hadoop-2.7.1/etc/hadoop 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:HBASE_LIBRARY_PATH=/home/aeris/hadoop-2.7.1/lib/native 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:HOSTNAME=r12s16.sjc.aristanetworks.com 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/home/aeris/hadoop-2.7.1/lib/native 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:SNAPPY_VERSION=1.1.1.7-1 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS= -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=11105 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:HBASE_VERSION=1.1.1 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:HADOOP_VERSION=2.7.1 2015-08-18 17:07:38,599 INFO [main] util.ServerCommandLine: env:PWD=/home/aeris/hbase-1.1.1 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS= -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=11103 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/home/aeris/hbase-1.1.1/bin/.. 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: env:HOME=/home/aeris 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: env:SHLVL=0 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: vmName=OpenJDK 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.45-b02 2015-08-18 17:07:38,600 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p, -Xmx10000m, -XX:+UseConcMarkSweepGC, -XX:NewSize=128m, -XX:MaxNewSize=128m, -XX:CMSInitiatingOccupancyFraction=75, -XX:+HeapDumpOnOutOfMemoryError, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -Djava.net.preferIPv4Stack=true, -Dsun.net.inetaddr.ttl=180, -Dproc_regionserver, -Xmx10g, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.port=11102, -Dhbase.log.dir=/data4/dfs/logs, -Dhbase.log.file=hbase.log, -Dhbase.home.dir=/home/aeris/hbase-1.1.1/bin/.., -Dhbase.id.str=, -Dhbase.root.logger=INFO,console, -Djava.library.path=/home/aeris/hadoop-2.7.1/lib/native, -Dhbase.security.logger=INFO,RFAS] 2015-08-18 17:07:38,817 INFO [main] regionserver.RSRpcServices: regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104 server-side HConnection retries=350 2015-08-18 17:07:38,953 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=5 2015-08-18 17:07:38,970 INFO [main] ipc.RpcServer: regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104: started 10 reader(s). 2015-08-18 17:07:39,039 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2015-08-18 17:07:39,057 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-08-18 17:07:39,057 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2015-08-18 17:07:39,480 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-08-18 17:07:39,623 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:9104 connecting to ZooKeeper ensemble=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=r12s16.sjc.aristanetworks.com 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_45-internal 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/aeris/hbase-1.1.1/bin/../conf:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/tools.jar:/home/aeris/hbase-1.1.1/bin/..:/home/aeris/hbase-1.1.1/bin/../lib/activation-1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/aopalliance-1.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/aeris/hbase-1.1.1/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/aeris/hbase-1.1.1/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/aeris/hbase-1.1.1/bin/../lib/api-util-1.0.0-M20.jar:/home/aeris/hbase-1.1.1/bin/../lib/asm-3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/avro-1.7.4.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-beanutils-1.7.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-cli-1.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-codec-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-collections-3.2.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-compress-1.4.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-configuration-1.6.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-daemon-1.0.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-digester-1.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-el-1.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-httpclient-3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-io-2.4.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-lang-2.6.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-logging-1.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-math-2.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-math3-3.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/commons-net-3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/disruptor-3.3.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/aeris/hbase-1.1.1/bin/../lib/guava-12.0.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/guice-3.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/guice-servlet-3.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-annotations-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-auth-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-client-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-hdfs-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-api-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-client-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-annotations-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-annotations-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-client-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-common-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-common-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-examples-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-hadoop-compat-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-hadoop2-compat-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-it-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-it-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-prefix-tree-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-procedure-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-protocol-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-rest-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-server-1.1.1-tests.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-server-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-shell-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/hbase-thrift-1.1.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/aeris/hbase-1.1.1/bin/../lib/httpclient-4.2.5.jar:/home/aeris/hbase-1.1.1/bin/../lib/httpcore-4.1.3.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-core-asl-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jackson-xc-1.9.13.jar:/home/aeris/hbase-1.1.1/bin/../lib/jamon-runtime-2.3.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/jasper-compiler-5.5.23.jar:/home/aeris/hbase-1.1.1/bin/../lib/jasper-runtime-5.5.23.jar:/home/aeris/hbase-1.1.1/bin/../lib/java-xmlbuilder-0.4.jar:/home/aeris/hbase-1.1.1/bin/../lib/javax.inject-1.jar:/home/aeris/hbase-1.1.1/bin/../lib/jaxb-api-2.2.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/aeris/hbase-1.1.1/bin/../lib/jcodings-1.0.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-client-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-core-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-guice-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-json-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jersey-server-1.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/jets3t-0.9.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/jettison-1.3.3.jar:/home/aeris/hbase-1.1.1/bin/../lib/jetty-6.1.26.jar:/home/aeris/hbase-1.1.1/bin/../lib/jetty-sslengine-6.1.26.jar:/home/aeris/hbase-1.1.1/bin/../lib/jetty-util-6.1.26.jar:/home/aeris/hbase-1.1.1/bin/../lib/joni-2.1.2.jar:/home/aeris/hbase-1.1.1/bin/../lib/jruby-complete-1.6.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsch-0.1.42.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsp-2.1-6.1.14.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/aeris/hbase-1.1.1/bin/../lib/jsr305-1.3.9.jar:/home/aeris/hbase-1.1.1/bin/../lib/junit-4.11.jar:/home/aeris/hbase-1.1.1/bin/../lib/leveldbjni-all-1.8.jar:/home/aeris/hbase-1.1.1/bin/../lib/libthrift-0.9.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/log4j-1.2.17.jar:/home/aeris/hbase-1.1.1/bin/../lib/metrics-core-2.2.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/netty-3.2.4.Final.jar:/home/aeris/hbase-1.1.1/bin/../lib/netty-all-4.0.23.Final.jar:/home/aeris/hbase-1.1.1/bin/../lib/paranamer-2.3.jar:/home/aeris/hbase-1.1.1/bin/../lib/protobuf-java-2.5.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/aeris/hbase-1.1.1/bin/../lib/servlet-api-2.5.jar:/home/aeris/hbase-1.1.1/bin/../lib/slf4j-api-1.7.7.jar:/home/aeris/hbase-1.1.1/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/aeris/hbase-1.1.1/bin/../lib/snappy-java-1.0.4.1.jar:/home/aeris/hbase-1.1.1/bin/../lib/spymemcached-2.11.6.jar:/home/aeris/hbase-1.1.1/bin/../lib/xmlenc-0.52.jar:/home/aeris/hbase-1.1.1/bin/../lib/xz-1.0.jar:/home/aeris/hbase-1.1.1/bin/../lib/zookeeper-3.4.6.jar::/home/aeris/hadoop-2.7.1/etc/hadoop 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/aeris/hadoop-2.7.1/lib/native 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2015-08-18 17:07:39,630 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2015-08-18 17:07:39,631 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2015-08-18 17:07:39,631 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=4.0.5 2015-08-18 17:07:39,631 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=aeris 2015-08-18 17:07:39,631 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/aeris 2015-08-18 17:07:39,631 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/aeris/hbase-1.1.1 2015-08-18 17:07:39,632 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181 sessionTimeout=90000 watcher=regionserver:91040x0, quorum=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181, baseZNode=/hbase 2015-08-18 17:07:39,651 INFO [main-SendThread(r12s5.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server r12s5.sjc.aristanetworks.com/172.24.32.5:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-18 17:07:39,656 INFO [main-SendThread(r12s5.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Socket connection established to r12s5.sjc.aristanetworks.com/172.24.32.5:2181, initiating session 2015-08-18 17:07:39,670 INFO [main-SendThread(r12s5.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server r12s5.sjc.aristanetworks.com/172.24.32.5:2181, sessionid = 0x24ec29a6c2b45ee, negotiated timeout = 40000 2015-08-18 17:07:39,684 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2015-08-18 17:07:39,684 INFO [RpcServer.listener,port=9104] ipc.RpcServer: RpcServer.listener,port=9104: starting 2015-08-18 17:07:39,745 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-08-18 17:07:39,748 INFO [main] http.HttpRequestLog: Http request log for http.requests.regionserver is not defined 2015-08-18 17:07:39,759 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2015-08-18 17:07:39,761 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2015-08-18 17:07:39,761 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-08-18 17:07:39,761 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2015-08-18 17:07:39,773 INFO [main] http.HttpServer: Jetty bound to port 9103 2015-08-18 17:07:39,773 INFO [main] mortbay.log: jetty-6.1.26 2015-08-18T17:07:39.815+0000: [CMS-concurrent-mark-start] 2015-08-18T17:07:39.819+0000: [CMS-concurrent-mark: 0.004/0.004 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 2015-08-18T17:07:39.819+0000: [CMS-concurrent-preclean-start] 2015-08-18T17:07:39.822+0000: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2015-08-18T17:07:39.835+0000: [CMS-concurrent-sweep-start] 2015-08-18T17:07:39.838+0000: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-18T17:07:39.838+0000: [CMS-concurrent-reset-start] 2015-08-18T17:07:39.883+0000: [CMS-concurrent-reset: 0.044/0.044 secs] [Times: user=0.13 sys=0.04, real=0.05 secs] 2015-08-18 17:07:40,071 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:9103 2015-08-18 17:07:40,122 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6be908f8 connecting to ZooKeeper ensemble=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181 2015-08-18 17:07:40,122 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] zookeeper.ZooKeeper: Initiating client connection, connectString=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181 sessionTimeout=90000 watcher=hconnection-0x6be908f80x0, quorum=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181, baseZNode=/hbase 2015-08-18 17:07:40,125 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-SendThread(r12s9.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server r12s9.sjc.aristanetworks.com/172.24.32.9:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-18 17:07:40,126 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-SendThread(r12s9.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Socket connection established to r12s9.sjc.aristanetworks.com/172.24.32.9:2181, initiating session 2015-08-18 17:07:40,135 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-SendThread(r12s9.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server r12s9.sjc.aristanetworks.com/172.24.32.9:2181, sessionid = 0x34ec1ac44b043fc, negotiated timeout = 40000 2015-08-18 17:07:40,192 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.HRegionServer: ClusterId : b6d96538-0c60-4d8c-85e9-5898dbfd16a8 2015-08-18 17:07:40,233 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.MemStoreFlusher: globalMemStoreLimit=4.0 G, globalMemStoreLimitLowMark=3.8 G, maxHeap=10.0 G 2015-08-18 17:07:40,237 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.HRegionServer: CompactionChecker runs every 10sec 2015-08-18 17:07:40,261 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled 2015-08-18 17:07:40,261 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled 2015-08-18 17:07:40,265 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.HRegionServer: reportForDuty to master=r12s3.sjc.aristanetworks.com,9102,1437811384604 with port=9104, startcode=1439917659071 2015-08-18 17:07:40,407 WARN [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-08-18 17:07:40,409 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] hfile.CacheConfig: Allocating LruBlockCache size=4.00 GB, blockSize=64 KB 2015-08-18 17:07:40,420 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=4399912, freeSize=4285219800, maxSize=4289619712, heapSize=4399912, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:07:40,497 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-08-18 17:07:40,565 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] wal.FSHLog: WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default, suffix=, logDir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071, archiveDir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs 2015-08-18 17:07:40,820 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] wal.FSHLog: Slow sync cost: 184 ms, current pipeline: [] 2015-08-18 17:07:40,821 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] wal.FSHLog: New WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439917660565 2015-08-18 17:07:40,845 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds 2015-08-18 17:07:40,884 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.ReplicationSourceManager: Current list of replicators: [r12s4.sjc.aristanetworks.com,9104,1439917656391, r12s7.sjc.aristanetworks.com,9104,1439917657031, r12s9.sjc.aristanetworks.com,9104,1439917657242, r12s16.sjc.aristanetworks.com,9104,1439917659071, r12s10.sjc.aristanetworks.com,9104,1439917657710, r12s12.sjc.aristanetworks.com,9104,1439917658042, r12s1.sjc.aristanetworks.com,9104,1439917654534, r12s6.sjc.aristanetworks.com,9104,1439917656836, r12s8.sjc.aristanetworks.com,9104,1439917484805, r12s5.sjc.aristanetworks.com,9104,1439917656631, r12s14.sjc.aristanetworks.com,9104,1439917658578, r12s3.sjc.aristanetworks.com,9104,1439917656128, r12s11.sjc.aristanetworks.com,9104,1439917657828, r12s13.sjc.aristanetworks.com,9104,1439917658339, r12s15.sjc.aristanetworks.com,9104,1439917658834, r12s2.sjc.aristanetworks.com,9104,1439917655725] other RSs: [r12s4.sjc.aristanetworks.com,9104,1439917656391, r12s7.sjc.aristanetworks.com,9104,1439917657031, r12s9.sjc.aristanetworks.com,9104,1439917657242, r12s16.sjc.aristanetworks.com,9104,1439917659071, r12s10.sjc.aristanetworks.com,9104,1439917657710, r12s12.sjc.aristanetworks.com,9104,1439917658042, r12s1.sjc.aristanetworks.com,9104,1439917654534, r12s6.sjc.aristanetworks.com,9104,1439917656836, r12s8.sjc.aristanetworks.com,9104,1439917484805, r12s5.sjc.aristanetworks.com,9104,1439917656631, r12s14.sjc.aristanetworks.com,9104,1439917658578, r12s3.sjc.aristanetworks.com,9104,1439917656128, r12s11.sjc.aristanetworks.com,9104,1439917657828, r12s13.sjc.aristanetworks.com,9104,1439917658339, r12s15.sjc.aristanetworks.com,9104,1439917658834, r12s2.sjc.aristanetworks.com,9104,1439917655725] 2015-08-18 17:07:40,910 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2cac047f connecting to ZooKeeper ensemble=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181 2015-08-18 17:07:40,910 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] zookeeper.ZooKeeper: Initiating client connection, connectString=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181 sessionTimeout=90000 watcher=hconnection-0x2cac047f0x0, quorum=0.zookeeper:2181,1.zookeeper:2181,2.zookeeper:2181, baseZNode=/hbase 2015-08-18 17:07:40,913 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-SendThread(r12s1.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server r12s1.sjc.aristanetworks.com/172.24.32.1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-18 17:07:40,913 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-SendThread(r12s1.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Socket connection established to r12s1.sjc.aristanetworks.com/172.24.32.1:2181, initiating session 2015-08-18 17:07:40,918 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-SendThread(r12s1.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server r12s1.sjc.aristanetworks.com/172.24.32.1:2181, sessionid = 0x14ec1ac4487454c, negotiated timeout = 40000 2015-08-18 17:07:40,953 INFO [SplitLogWorker-r12s16:9104] regionserver.SplitLogWorker: SplitLogWorker r12s16.sjc.aristanetworks.com,9104,1439917659071 starting 2015-08-18 17:07:40,955 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.HeapMemoryManager: Starting HeapMemoryTuner chore. 2015-08-18 17:07:40,958 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] regionserver.HRegionServer: Serving as r12s16.sjc.aristanetworks.com,9104,1439917659071, RpcServer on r12s16.sjc.aristanetworks.com/172.24.32.16:9104, sessionid=0x24ec29a6c2b45ee 2015-08-18 17:07:40,968 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104] quotas.RegionServerQuotaManager: Quota support disabled 2015-08-18 17:08:19,613 INFO [PriorityRpcServer.handler=1,queue=1,port=9104] regionserver.RSRpcServices: Open aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 17:08:19,748 INFO [RS_OPEN_REGION-r12s16:9104-0] compress.CodecPool: Got brand-new compressor [.snappy] 2015-08-18 17:08:19,799 INFO [StoreOpener-4f9970eda1177fedb60b808a938a6c7d-1] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=4399912, freeSize=4285219800, maxSize=4289619712, heapSize=4399912, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:08:19,806 INFO [StoreOpener-4f9970eda1177fedb60b808a938a6c7d-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-08-18 17:08:19,859 INFO [StoreFileOpenerThread-e-1] compress.CodecPool: Got brand-new decompressor [.snappy] 2015-08-18 17:08:20,289 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HRegion: Onlined 4f9970eda1177fedb60b808a938a6c7d; next sequenceid=68833024 2015-08-18 17:08:20,314 INFO [PostOpenDeployTasks:4f9970eda1177fedb60b808a938a6c7d] regionserver.HRegionServer: Post open deploy tasks for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 17:08:20,333 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 17:08:20,334 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.HStore: Starting compaction of 9 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=4.9 G 2015-08-18 17:08:20,414 INFO [PostOpenDeployTasks:4f9970eda1177fedb60b808a938a6c7d] hbase.MetaTableAccessor: Updated row aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. with server=r12s16.sjc.aristanetworks.com,9104,1439917659071 2015-08-18 17:08:20,913 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=9, currentSize=4705080, freeSize=4284914632, maxSize=4289619712, heapSize=4705080, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:08:21,044 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] compress.CodecPool: Got brand-new compressor [.snappy] 2015-08-18 17:08:29,432 INFO [PriorityRpcServer.handler=0,queue=0,port=9104] regionserver.RSRpcServices: Open aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 17:08:29,485 INFO [StoreOpener-4691fd4d7548ec013bb9d0c89f18a5f3-1] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=9, currentSize=4705080, freeSize=4284914632, maxSize=4289619712, heapSize=4705080, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:08:29,485 INFO [StoreOpener-4691fd4d7548ec013bb9d0c89f18a5f3-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-08-18 17:08:29,497 INFO [StoreFileOpenerThread-e-1] compress.CodecPool: Got brand-new decompressor [.snappy] 2015-08-18 17:08:30,058 INFO [RS_OPEN_REGION-r12s16:9104-1] regionserver.HRegion: Onlined 4691fd4d7548ec013bb9d0c89f18a5f3; next sequenceid=67412319 2015-08-18 17:08:30,062 INFO [PostOpenDeployTasks:4691fd4d7548ec013bb9d0c89f18a5f3] regionserver.HRegionServer: Post open deploy tasks for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 17:08:30,068 INFO [PostOpenDeployTasks:4691fd4d7548ec013bb9d0c89f18a5f3] hbase.MetaTableAccessor: Updated row aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. with server=r12s16.sjc.aristanetworks.com,9104,1439917659071 2015-08-18 17:08:30,069 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 17:08:30,070 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 5 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=78.7 M 2015-08-18 17:08:30,084 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=9, currentSize=4705080, freeSize=4284914632, maxSize=4289619712, heapSize=4705080, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:08:30,192 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] compress.CodecPool: Got brand-new compressor [.snappy] 2015-08-18 17:08:30,192 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] compress.CodecPool: Got brand-new compressor [.snappy] 2015-08-18 17:08:32,644 INFO [PriorityRpcServer.handler=3,queue=1,port=9104] regionserver.RSRpcServices: Open aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 17:08:32,664 INFO [StoreOpener-ff398ccd85feb23e75c0c4ce80e74cbe-1] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=15, currentSize=4926104, freeSize=4284693608, maxSize=4289619712, heapSize=4926104, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:08:32,665 INFO [StoreOpener-ff398ccd85feb23e75c0c4ce80e74cbe-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-08-18 17:08:33,025 INFO [RS_OPEN_REGION-r12s16:9104-2] regionserver.HRegion: Onlined ff398ccd85feb23e75c0c4ce80e74cbe; next sequenceid=60733765 2015-08-18 17:08:33,028 INFO [PostOpenDeployTasks:ff398ccd85feb23e75c0c4ce80e74cbe] regionserver.HRegionServer: Post open deploy tasks for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 17:08:33,032 INFO [PostOpenDeployTasks:ff398ccd85feb23e75c0c4ce80e74cbe] hbase.MetaTableAccessor: Updated row aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. with server=r12s16.sjc.aristanetworks.com,9104,1439917659071 2015-08-18 17:08:33,190 INFO [B.defaultRpcServer.handler=21,queue=1,port=9104] compress.CodecPool: Got brand-new decompressor [.snappy] 2015-08-18 17:08:38,151 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 5 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into b84d2a53e1494fb38bf72fc44265d7f7(size=78.5 M), total size for store is 4.7 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-18 17:08:38,153 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=5, fileSize=78.7 M, priority=18, time=3454838480698851; duration=8sec 2015-08-18 17:08:38,160 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 17:08:38,160 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 4 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=30.1 M 2015-08-18 17:08:38,174 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=18, currentSize=5048912, freeSize=4284570800, maxSize=4289619712, heapSize=5048912, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:08:42,629 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 4 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into b78385d4f6cd430d90680abb3b48fac2(size=29.9 M), total size for store is 3.0 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-18 17:08:42,629 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=4, fileSize=30.1 M, priority=17, time=3454846570596299; duration=4sec .... 2015-08-18 17:12:40,421 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.78 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=15, accesses=510537, hits=21, hitRatio=0.00%, , cachingAccesses=51, cachingHits=21, cachingHitsRatio=41.18%, evictions=29, evicted=7, evictedPerRun=0.24137930572032928 2015-08-18 17:14:49,259 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.HStore: Completed major compaction of 9 (all) file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 23eee31d8ff6466cb814dc04d57b5be5(size=4.4 G), total size for store is 4.4 G. This selection was in queue for 0sec, and took 6mins, 28sec to execute. 2015-08-18 17:14:49,259 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=9, fileSize=4.9 G, priority=16, time=3454828743720178; duration=6mins, 28sec 2015-08-18 17:14:53,418 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439917660565 with entries=48307, filesize=121.61 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918093353 2015-08-18 17:17:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.49 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=6, accesses=761224, hits=21, hitRatio=0.00%, , cachingAccesses=51, cachingHits=21, cachingHitsRatio=41.18%, evictions=59, evicted=16, evictedPerRun=0.2711864411830902 2015-08-18 17:18:32,966 INFO [sync.4] wal.FSHLog: Slow sync cost: 117 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.1:10110] 2015-08-18 17:18:32,966 INFO [sync.0] wal.FSHLog: Slow sync cost: 103 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.1:10110] 2015-08-18 17:20:58,439 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:20:59,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=68898365, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/cc68541985e44cdfaa06377461e6de6a 2015-08-18 17:20:59,410 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/cc68541985e44cdfaa06377461e6de6a, entries=362694, sequenceid=68898365, filesize=8.8 M 2015-08-18 17:20:59,412 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218304, currentsize=147.51 KB/151048 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 972ms, sequenceid=68898365, compaction requested=false 2015-08-18 17:21:01,114 INFO [sync.3] wal.FSHLog: Slow sync cost: 129 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.1:10110] 2015-08-18 17:21:01,114 INFO [sync.4] wal.FSHLog: Slow sync cost: 112 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.1:10110] 2015-08-18 17:21:01,114 INFO [sync.0] wal.FSHLog: Slow sync cost: 106 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.1:10110] 2015-08-18 17:21:27,960 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918093353 with entries=50254, filesize=121.61 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918487872 2015-08-18 17:21:53,700 WARN [ResponseProcessor for block BP-1466258523-172.24.32.1-1437768622582:blk_1073789416_48903] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1466258523-172.24.32.1-1437768622582:blk_1073789416_48903 java.io.IOException: Bad response ERROR for block BP-1466258523-172.24.32.1-1437768622582:blk_1073789416_48903 from datanode 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828) 2015-08-18 17:21:53,703 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918487872 block BP-1466258523-172.24.32.1-1437768622582:blk_1073789416_48903] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073789416_48903 in pipeline 172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.9:10110: bad datanode 172.24.32.8:10110 2015-08-18 17:21:53,870 INFO [sync.4] wal.FSHLog: Slow sync cost: 121 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.5:10110] 2015-08-18 17:22:36,032 INFO [main-EventThread] replication.ReplicationTrackerZKImpl: /hbase/rs/r12s8.sjc.aristanetworks.com,9104,1439917484805 znode expired, triggering replicatorRemoved event 2015-08-18 17:22:36,156 INFO [SplitLogWorker-r12s16:9104] coordination.ZkSplitLogWorkerCoordination: worker r12s16.sjc.aristanetworks.com,9104,1439917659071 acquired task /hbase/splitWAL/WALs%2Fr12s8.sjc.aristanetworks.com%2C9104%2C1439917484805-splitting%2Fr12s8.sjc.aristanetworks.com%252C9104%252C1439917484805.default.1439917486905 2015-08-18 17:22:36,181 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] wal.WALSplitter: Splitting wal: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s8.sjc.aristanetworks.com,9104,1439917484805-splitting/r12s8.sjc.aristanetworks.com%2C9104%2C1439917484805.default.1439917486905, length=129503507 2015-08-18 17:22:36,181 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] wal.WALSplitter: DistributedLogReplay = false 2015-08-18 17:22:36,191 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] util.FSHDFSUtils: Recovering lease on dfs file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s8.sjc.aristanetworks.com,9104,1439917484805-splitting/r12s8.sjc.aristanetworks.com%2C9104%2C1439917484805.default.1439917486905 2015-08-18 17:22:36,196 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] util.FSHDFSUtils: recoverLease=true, attempt=0 on file=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s8.sjc.aristanetworks.com,9104,1439917484805-splitting/r12s8.sjc.aristanetworks.com%2C9104%2C1439917484805.default.1439917486905 after 5ms 2015-08-18 17:22:36,275 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/326031e759754e896de910ffc9ae8762/recovered.edits/0000000000039373916.temp region=326031e759754e896de910ffc9ae8762 2015-08-18 17:22:36,283 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ca44c199d6140066c8bd3853cce05d01/recovered.edits/0000000000016730456.temp region=ca44c199d6140066c8bd3853cce05d01 2015-08-18 17:22:36,284 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6ca4c4f9ff8b65db4781c05c274f7d64/recovered.edits/0000000000016285003.temp region=6ca4c4f9ff8b65db4781c05c274f7d64 2015-08-18 17:22:36,317 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/recovered.edits/0000000000060721599.temp region=ff398ccd85feb23e75c0c4ce80e74cbe 2015-08-18 17:22:36,317 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/recovered.edits/0000000000067399165.temp region=4691fd4d7548ec013bb9d0c89f18a5f3 2015-08-18 17:22:36,325 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/21c635c1e9aaf404f2b5918e3a930fa3/recovered.edits/0000000000066275954.temp region=21c635c1e9aaf404f2b5918e3a930fa3 2015-08-18 17:22:36,328 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: This region's directory doesn't exist: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/afe7a0c72a42636a960f51dbde7a60b5. It is very likely that it was already split so it's safe to discard those edits. 2015-08-18 17:22:36,329 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: This region's directory doesn't exist: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7df67512d33372ab857cbf31adaef1ac. It is very likely that it was already split so it's safe to discard those edits. 2015-08-18 17:22:36,350 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/fe17e46799878a25bba31be81c4e10a8/recovered.edits/0000000000069655546.temp region=fe17e46799878a25bba31be81c4e10a8 2015-08-18 17:22:36,358 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/03310958fed9f3c667184c9cc4df8973/recovered.edits/0000000000070214676.temp region=03310958fed9f3c667184c9cc4df8973 2015-08-18 17:22:36,358 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/bf352d76874ff4be669d6517510f30f0/recovered.edits/0000000000062862662.temp region=bf352d76874ff4be669d6517510f30f0 2015-08-18 17:22:36,360 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: This region's directory doesn't exist: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/433bfa51632f8ecdc86906c9db6e26ab. It is very likely that it was already split so it's safe to discard those edits. 2015-08-18 17:22:36,361 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: This region's directory doesn't exist: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e91a08976eb522fc4e72e1c6f4d05da4. It is very likely that it was already split so it's safe to discard those edits. 2015-08-18 17:22:36,367 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/b172684d7bab2f0f162177eebea9266f/recovered.edits/0000000000064891717.temp region=b172684d7bab2f0f162177eebea9266f 2015-08-18 17:22:36,369 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: This region's directory doesn't exist: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/88f0d88dbca36c6bc15829258f76d08c. It is very likely that it was already split so it's safe to discard those edits. 2015-08-18 17:22:36,460 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2729bf98e430953c6be2bbba3719de46/recovered.edits/0000000000057061516.temp region=2729bf98e430953c6be2bbba3719de46 2015-08-18 17:22:36,460 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/047c4543e90c47d401b91f9719b871a9/recovered.edits/0000000000066786540.temp region=047c4543e90c47d401b91f9719b871a9 2015-08-18 17:22:36,502 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/8b98fcde2e7e9aeb24b39f522fb2c5b1/recovered.edits/0000000000060840570.temp region=8b98fcde2e7e9aeb24b39f522fb2c5b1 2015-08-18 17:22:36,542 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/71dd522dced376cc0a2a7f94c8002b4e/recovered.edits/0000000000051692512.temp region=71dd522dced376cc0a2a7f94c8002b4e 2015-08-18 17:22:36,542 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/recovered.edits/0000000000047807195.temp region=a1233a97ab54c0f50ac9a39cd05f75a6 2015-08-18 17:22:36,551 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/493b2a8b54ed4ce5e957d7ebc4954094/recovered.edits/0000000000043718033.temp region=493b2a8b54ed4ce5e957d7ebc4954094 2015-08-18 17:22:36,567 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/3bd458557d5138f71f915ad0bb018379/recovered.edits/0000000000048632633.temp region=3bd458557d5138f71f915ad0bb018379 2015-08-18 17:22:36,567 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e3816977aed173331e83980c1d1ab8df/recovered.edits/0000000000044626964.temp region=e3816977aed173331e83980c1d1ab8df 2015-08-18 17:22:36,576 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris_devices/4910f068a4bf2f2680ea5947e55e61b3/recovered.edits/0000000000247319232.temp region=4910f068a4bf2f2680ea5947e55e61b3 2015-08-18 17:22:36,592 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/49ace374826589d77ff9a582ea04ea4b/recovered.edits/0000000000039558334.temp region=49ace374826589d77ff9a582ea04ea4b 2015-08-18 17:22:36,592 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/96749cd94e0daf258dc9c44852699662/recovered.edits/0000000000036484743.temp region=96749cd94e0daf258dc9c44852699662 2015-08-18 17:22:36,601 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2dbaa97a366ed76d55a7fdf42e6d17d1/recovered.edits/0000000000013839051.temp region=2dbaa97a366ed76d55a7fdf42e6d17d1 2015-08-18 17:22:36,617 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2faa5f939d8dc8b74e7d280c432e4a7a/recovered.edits/0000000000009611142.temp region=2faa5f939d8dc8b74e7d280c432e4a7a 2015-08-18 17:22:36,617 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e9e750725bc592463c79037d6136c198/recovered.edits/0000000000018045556.temp region=e9e750725bc592463c79037d6136c198 2015-08-18 17:22:36,625 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/43cf05b07f948ba78b55d57e8294418c/recovered.edits/0000000000019709620.temp region=43cf05b07f948ba78b55d57e8294418c 2015-08-18 17:22:36,642 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/41211de6ffbef148229a010396832c65/recovered.edits/0000000000015989631.temp region=41211de6ffbef148229a010396832c65 2015-08-18 17:22:36,642 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/bc8a00d2663233dc478b28c3f0018b44/recovered.edits/0000000000015935968.temp region=bc8a00d2663233dc478b28c3f0018b44 2015-08-18 17:22:36,648 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: This region's directory doesn't exist: hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/107ccb22de210ba16abb96f4569359af. It is very likely that it was already split so it's safe to discard those edits. 2015-08-18 17:22:36,650 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e0edf670377707b98006796bb59ab47/recovered.edits/0000000000012215628.temp region=7e0edf670377707b98006796bb59ab47 2015-08-18 17:22:36,667 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-0] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/hbase/namespace/ce9cd54c0b4b91a0eacb72cd59423d49/recovered.edits/0000000000000000017.temp region=ce9cd54c0b4b91a0eacb72cd59423d49 2015-08-18 17:22:36,667 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-2] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/727c24b915f6ac9bd1a01d915feb82ee/recovered.edits/0000000000013811644.temp region=727c24b915f6ac9bd1a01d915feb82ee 2015-08-18 17:22:36,668 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0-Writer-1] wal.WALSplitter: Creating writer path=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4aba6bd7e193382b3fb268f69814e78e/recovered.edits/0000000000011737304.temp region=4aba6bd7e193382b3fb268f69814e78e 2015-08-18 17:22:37,505 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] wal.WALSplitter: Split writers finished 2015-08-18 17:22:37,567 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/21c635c1e9aaf404f2b5918e3a930fa3/recovered.edits/0000000000066275954.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/21c635c1e9aaf404f2b5918e3a930fa3/recovered.edits/0000000000066275954 2015-08-18 17:22:37,567 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/047c4543e90c47d401b91f9719b871a9/recovered.edits/0000000000066786540.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/047c4543e90c47d401b91f9719b871a9/recovered.edits/0000000000066786540 2015-08-18 17:22:37,575 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/03310958fed9f3c667184c9cc4df8973/recovered.edits/0000000000070214676.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/03310958fed9f3c667184c9cc4df8973/recovered.edits/0000000000070214676 2015-08-18 17:22:37,616 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2dbaa97a366ed76d55a7fdf42e6d17d1/recovered.edits/0000000000013839051.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2dbaa97a366ed76d55a7fdf42e6d17d1/recovered.edits/0000000000013839051 2015-08-18 17:22:37,616 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2729bf98e430953c6be2bbba3719de46/recovered.edits/0000000000057061516.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2729bf98e430953c6be2bbba3719de46/recovered.edits/0000000000057061516 2015-08-18 17:22:37,625 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2faa5f939d8dc8b74e7d280c432e4a7a/recovered.edits/0000000000009611142.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/2faa5f939d8dc8b74e7d280c432e4a7a/recovered.edits/0000000000009611142 2015-08-18 17:22:37,666 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/326031e759754e896de910ffc9ae8762/recovered.edits/0000000000039373916.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/326031e759754e896de910ffc9ae8762/recovered.edits/0000000000039373916 2015-08-18 17:22:37,666 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/3bd458557d5138f71f915ad0bb018379/recovered.edits/0000000000048632633.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/3bd458557d5138f71f915ad0bb018379/recovered.edits/0000000000048632633 2015-08-18 17:22:37,675 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/41211de6ffbef148229a010396832c65/recovered.edits/0000000000015989631.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/41211de6ffbef148229a010396832c65/recovered.edits/0000000000015989631 2015-08-18 17:22:37,716 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/recovered.edits/0000000000067399165.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/recovered.edits/0000000000067399165 2015-08-18 17:22:37,725 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/43cf05b07f948ba78b55d57e8294418c/recovered.edits/0000000000019709620.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/43cf05b07f948ba78b55d57e8294418c/recovered.edits/0000000000019709620 2015-08-18 17:22:37,725 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris_devices/4910f068a4bf2f2680ea5947e55e61b3/recovered.edits/0000000000247319232.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris_devices/4910f068a4bf2f2680ea5947e55e61b3/recovered.edits/0000000000247319232 2015-08-18 17:22:37,746 INFO [Thread-200] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:22:37,746 INFO [Thread-200] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789459_48949 2015-08-18 17:22:37,760 INFO [Thread-200] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:22:37,775 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/493b2a8b54ed4ce5e957d7ebc4954094/recovered.edits/0000000000043718033.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/493b2a8b54ed4ce5e957d7ebc4954094/recovered.edits/0000000000043718033 2015-08-18 17:22:37,775 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4aba6bd7e193382b3fb268f69814e78e/recovered.edits/0000000000011737304.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4aba6bd7e193382b3fb268f69814e78e/recovered.edits/0000000000011737304 2015-08-18 17:22:37,808 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6ca4c4f9ff8b65db4781c05c274f7d64/recovered.edits/0000000000016285003.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6ca4c4f9ff8b65db4781c05c274f7d64/recovered.edits/0000000000016285145 2015-08-18 17:22:37,808 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/49ace374826589d77ff9a582ea04ea4b/recovered.edits/0000000000039558334.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/49ace374826589d77ff9a582ea04ea4b/recovered.edits/0000000000039558334 2015-08-18 17:22:37,825 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/71dd522dced376cc0a2a7f94c8002b4e/recovered.edits/0000000000051692512.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/71dd522dced376cc0a2a7f94c8002b4e/recovered.edits/0000000000051692512 2015-08-18 17:22:37,850 INFO [Thread-193] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:22:37,850 INFO [Thread-193] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789474_48964 2015-08-18 17:22:37,858 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e0edf670377707b98006796bb59ab47/recovered.edits/0000000000012215628.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e0edf670377707b98006796bb59ab47/recovered.edits/0000000000012215628 2015-08-18 17:22:37,866 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/727c24b915f6ac9bd1a01d915feb82ee/recovered.edits/0000000000013811644.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/727c24b915f6ac9bd1a01d915feb82ee/recovered.edits/0000000000013811644 2015-08-18 17:22:37,867 INFO [Thread-193] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:22:37,975 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/96749cd94e0daf258dc9c44852699662/recovered.edits/0000000000036484743.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/96749cd94e0daf258dc9c44852699662/recovered.edits/0000000000036484743 2015-08-18 17:22:38,050 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/8b98fcde2e7e9aeb24b39f522fb2c5b1/recovered.edits/0000000000060840570.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/8b98fcde2e7e9aeb24b39f522fb2c5b1/recovered.edits/0000000000060840570 2015-08-18 17:22:38,050 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/recovered.edits/0000000000047807195.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/recovered.edits/0000000000047807195 2015-08-18 17:22:38,069 INFO [Thread-188] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:22:38,069 INFO [Thread-206] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:22:38,070 INFO [Thread-188] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789488_48978 2015-08-18 17:22:38,070 INFO [Thread-206] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789487_48977 2015-08-18 17:22:38,117 INFO [Thread-206] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:22:38,117 INFO [Thread-188] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:22:38,200 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/b172684d7bab2f0f162177eebea9266f/recovered.edits/0000000000064891717.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/b172684d7bab2f0f162177eebea9266f/recovered.edits/0000000000064891717 2015-08-18 17:22:38,300 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/bc8a00d2663233dc478b28c3f0018b44/recovered.edits/0000000000015935968.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/bc8a00d2663233dc478b28c3f0018b44/recovered.edits/0000000000015935968 2015-08-18 17:22:38,300 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/bf352d76874ff4be669d6517510f30f0/recovered.edits/0000000000062862662.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/bf352d76874ff4be669d6517510f30f0/recovered.edits/0000000000062862662 2015-08-18 17:22:38,308 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ca44c199d6140066c8bd3853cce05d01/recovered.edits/0000000000016730456.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ca44c199d6140066c8bd3853cce05d01/recovered.edits/0000000000016730456 2015-08-18 17:22:38,330 INFO [Thread-204] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:22:38,330 INFO [Thread-204] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789505_48995 2015-08-18 17:22:38,408 INFO [Thread-204] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:22:38,492 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e3816977aed173331e83980c1d1ab8df/recovered.edits/0000000000044626964.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e3816977aed173331e83980c1d1ab8df/recovered.edits/0000000000044626964 2015-08-18 17:22:38,492 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/hbase/namespace/ce9cd54c0b4b91a0eacb72cd59423d49/recovered.edits/0000000000000000017.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/hbase/namespace/ce9cd54c0b4b91a0eacb72cd59423d49/recovered.edits/0000000000000000017 2015-08-18 17:22:38,600 INFO [split-log-closeStream-2] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e9e750725bc592463c79037d6136c198/recovered.edits/0000000000018045556.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/e9e750725bc592463c79037d6136c198/recovered.edits/0000000000018045556 2015-08-18 17:22:38,676 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/recovered.edits/0000000000060721599.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/recovered.edits/0000000000060721599 2015-08-18 17:22:38,676 INFO [split-log-closeStream-3] wal.WALSplitter: Rename hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/fe17e46799878a25bba31be81c4e10a8/recovered.edits/0000000000069655546.temp to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/fe17e46799878a25bba31be81c4e10a8/recovered.edits/0000000000069655546 2015-08-18 17:22:38,676 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] wal.WALSplitter: Processed 13719 edits across 31 regions; edits skipped=5170; log file=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s8.sjc.aristanetworks.com,9104,1439917484805-splitting/r12s8.sjc.aristanetworks.com%2C9104%2C1439917484805.default.1439917486905, length=129503507, corrupted=false, progress failed=false 2015-08-18 17:22:38,699 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] coordination.ZkSplitLogWorkerCoordination: successfully transitioned task /hbase/splitWAL/WALs%2Fr12s8.sjc.aristanetworks.com%2C9104%2C1439917484805-splitting%2Fr12s8.sjc.aristanetworks.com%252C9104%252C1439917484805.default.1439917486905 to final state DONE r12s16.sjc.aristanetworks.com,9104,1439917659071 2015-08-18 17:22:38,699 INFO [RS_LOG_REPLAY_OPS-r12s16:9104-0] handler.WALSplitterHandler: worker r12s16.sjc.aristanetworks.com,9104,1439917659071 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@734e2ff8 in 2539ms 2015-08-18 17:22:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.49 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=6, accesses=761224, hits=21, hitRatio=0.00%, , cachingAccesses=51, cachingHits=21, cachingHitsRatio=41.18%, evictions=89, evicted=16, evictedPerRun=0.17977528274059296 2015-08-18 17:22:48,651 INFO [PriorityRpcServer.handler=2,queue=0,port=9104] regionserver.RSRpcServices: Open aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 17:22:48,678 INFO [StoreOpener-7e981f7e8b0732d7dfcd2e63005063c7-1] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=6, currentSize=4702880, freeSize=4284916832, maxSize=4289619712, heapSize=4702880, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:22:48,679 INFO [StoreOpener-7e981f7e8b0732d7dfcd2e63005063c7-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-08-18 17:22:48,712 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HRegion: Replaying edits from hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/recovered.edits/0000000000066861810 2015-08-18 17:22:48,881 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HRegion: Replaying edits from hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/recovered.edits/0000000000066893582 2015-08-18 17:22:50,176 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 73.02 MB, and 1/1 column families' memstores are being flushed.; wal is null, using passed sequenceid=66893582 2015-08-18 17:22:50,644 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=66893582, memsize=73.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/07e0dbc6fdcf4d09a8620cb3d21fdf2d 2015-08-18 17:22:50,664 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/07e0dbc6fdcf4d09a8620cb3d21fdf2d, entries=198030, sequenceid=66893582, filesize=5.0 M 2015-08-18 17:22:50,664 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HRegion: Finished memstore flush of ~73.02 MB/76569616, currentsize=0 B/0 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 489ms, sequenceid=66893582, compaction requested=true; wal=null 2015-08-18 17:22:50,735 INFO [RS_OPEN_REGION-r12s16:9104-0] regionserver.HRegion: Onlined 7e981f7e8b0732d7dfcd2e63005063c7; next sequenceid=66893583 2015-08-18 17:22:50,738 INFO [PostOpenDeployTasks:7e981f7e8b0732d7dfcd2e63005063c7] regionserver.HRegionServer: Post open deploy tasks for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 17:22:50,741 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 17:22:50,742 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 4 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp, totalSize=430.7 M 2015-08-18 17:22:50,745 INFO [PostOpenDeployTasks:7e981f7e8b0732d7dfcd2e63005063c7] hbase.MetaTableAccessor: Updated row aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. with server=r12s16.sjc.aristanetworks.com,9104,1439917659071 2015-08-18 17:22:50,753 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=6, currentSize=4702880, freeSize=4284916832, maxSize=4289619712, heapSize=4702880, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:22:53,510 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:22:54,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67481682, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/78b28b0ac1cd4486a81e3f5cd9383d14 2015-08-18 17:22:54,365 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/78b28b0ac1cd4486a81e3f5cd9383d14, entries=386823, sequenceid=67481682, filesize=9.0 M 2015-08-18 17:22:54,366 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221328, currentsize=1.08 MB/1136816 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 856ms, sequenceid=67481682, compaction requested=true 2015-08-18 17:23:16,087 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Atomically moving r12s8.sjc.aristanetworks.com,9104,1439917484805's wals to my queue 2015-08-18 17:23:20,624 INFO [DataStreamer for file /hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/8adf97e88b034a9397899c0c9b424c73] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:23:20,624 INFO [DataStreamer for file /hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/8adf97e88b034a9397899c0c9b424c73] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789681_49171 2015-08-18 17:23:20,644 INFO [DataStreamer for file /hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/8adf97e88b034a9397899c0c9b424c73] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:23:24,414 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 4 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into 8adf97e88b034a9397899c0c9b424c73(size=431.0 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 33sec to execute. 2015-08-18 17:23:24,414 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., storeName=e, fileCount=4, fileSize=430.7 M, priority=20, time=3455699152527243; duration=33sec 2015-08-18 17:23:50,051 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:23:50,825 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=60803026, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/07988df848fe41e2a707e06d390cfa09 2015-08-18 17:23:50,846 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/07988df848fe41e2a707e06d390cfa09, entries=367703, sequenceid=60803026, filesize=8.8 M 2015-08-18 17:23:50,847 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134224024, currentsize=82.98 KB/84968 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 797ms, sequenceid=60803026, compaction requested=true 2015-08-18 17:26:40,244 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918487872 with entries=41762, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918800070 2015-08-18 17:26:40,246 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439917660565 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439917660565 2015-08-18 17:27:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.61 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=7, accesses=835716, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=119, evicted=20, evictedPerRun=0.16806723177433014 2015-08-18T17:28:06.620+0000: [CMS-concurrent-mark-start] 2015-08-18T17:28:06.678+0000: [CMS-concurrent-mark: 0.058/0.058 secs] [Times: user=0.29 sys=0.00, real=0.06 secs] 2015-08-18T17:28:06.678+0000: [CMS-concurrent-preclean-start] 2015-08-18T17:28:06.681+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-18T17:28:06.681+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18T17:28:09.641+0000: [CMS-concurrent-abortable-preclean: 0.856/2.960 secs] [Times: user=1.09 sys=0.06, real=2.96 secs] 2015-08-18T17:28:09.682+0000: [CMS-concurrent-sweep-start] 2015-08-18T17:28:09.990+0000: [CMS-concurrent-sweep: 0.308/0.308 secs] [Times: user=0.33 sys=0.01, real=0.30 secs] 2015-08-18T17:28:09.990+0000: [CMS-concurrent-reset-start] 2015-08-18T17:28:10.010+0000: [CMS-concurrent-reset: 0.020/0.020 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-18 17:31:26,385 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918800070 with entries=47715, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919086213 2015-08-18 17:32:08,033 INFO [sync.2] wal.FSHLog: Slow sync cost: 131 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:32:08,033 INFO [sync.3] wal.FSHLog: Slow sync cost: 124 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:32:08,033 INFO [sync.0] wal.FSHLog: Slow sync cost: 105 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:32:08,033 INFO [sync.4] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:32:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.61 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=7, accesses=835716, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=149, evicted=20, evictedPerRun=0.1342281848192215 2015-08-18 17:34:00,829 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:34:01,601 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=68960942, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/ed897c5ee74c4c20bc5a9e6ecba93350 2015-08-18 17:34:01,638 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/ed897c5ee74c4c20bc5a9e6ecba93350, entries=363770, sequenceid=68960942, filesize=8.8 M 2015-08-18 17:34:01,639 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134227648, currentsize=145.38 KB/148864 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 810ms, sequenceid=68960942, compaction requested=true 2015-08-18 17:34:52,127 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:34:52,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=66954164, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/38ee279bcad749f39f2a758e3ed1ae24 2015-08-18 17:34:52,919 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/38ee279bcad749f39f2a758e3ed1ae24, entries=347378, sequenceid=66954164, filesize=8.7 M 2015-08-18 17:34:52,920 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134227912, currentsize=76.63 KB/78464 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 793ms, sequenceid=66954164, compaction requested=true 2015-08-18 17:36:13,533 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919086213 with entries=47225, filesize=121.74 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919373384 2015-08-18 17:36:13,535 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918093353 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918093353 2015-08-18 17:36:14,151 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:36:14,957 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67549287, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/d2ab64bbb91a429892e5196a44ce307a 2015-08-18 17:36:14,994 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/d2ab64bbb91a429892e5196a44ce307a, entries=387223, sequenceid=67549287, filesize=9.0 M 2015-08-18 17:36:14,995 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222144, currentsize=72.66 KB/74408 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 844ms, sequenceid=67549287, compaction requested=true 2015-08-18 17:36:14,996 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 17:36:14,997 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=96.5 M 2015-08-18 17:36:15,003 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=7, currentSize=4834632, freeSize=4284785080, maxSize=4289619712, heapSize=4834632, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:36:22,875 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into cb1405636d5d4087a06bd4b58096203d(size=96.3 M), total size for store is 4.7 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-18 17:36:22,875 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=96.5 M, priority=20, time=3456503407886451; duration=7sec 2015-08-18 17:37:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.61 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=7, accesses=850001, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=179, evicted=20, evictedPerRun=0.11173184216022491 2015-08-18 17:38:55,243 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:38:55,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=60873503, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/2eaecaf83de24f30b5d572d12ff36d34 2015-08-18 17:38:56,012 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/2eaecaf83de24f30b5d572d12ff36d34, entries=367321, sequenceid=60873503, filesize=8.8 M 2015-08-18 17:38:56,013 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218008, currentsize=100.03 KB/102432 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 770ms, sequenceid=60873503, compaction requested=true 2015-08-18 17:38:56,016 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 17:38:56,017 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=47.6 M 2015-08-18 17:38:56,021 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=7, currentSize=4834632, freeSize=4284785080, maxSize=4289619712, heapSize=4834632, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:38:56,060 INFO [Thread-306] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:38:56,060 INFO [Thread-306] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789731_49221 2015-08-18 17:38:56,075 INFO [Thread-306] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:38:59,992 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 79f229701057484a8516ee37eb27bfbb(size=47.4 M), total size for store is 3.1 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-18 17:38:59,993 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=47.6 M, priority=18, time=3456664427494391; duration=3sec 2015-08-18 17:40:58,814 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919373384 with entries=48118, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919658729 2015-08-18 17:40:58,815 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918487872 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918487872 2015-08-18 17:40:58,847 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918800070 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439918800070 2015-08-18 17:41:04,016 INFO [sync.1] wal.FSHLog: Slow sync cost: 205 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.5:10110] 2015-08-18 17:41:04,016 INFO [sync.2] wal.FSHLog: Slow sync cost: 194 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.5:10110] 2015-08-18 17:41:04,016 INFO [sync.4] wal.FSHLog: Slow sync cost: 128 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.5:10110] 2015-08-18 17:41:04,016 INFO [sync.3] wal.FSHLog: Slow sync cost: 188 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.5:10110] 2015-08-18T17:41:52.735+0000: [CMS-concurrent-mark-start] 2015-08-18T17:41:52.795+0000: [CMS-concurrent-mark: 0.060/0.060 secs] [Times: user=0.30 sys=0.00, real=0.06 secs] 2015-08-18T17:41:52.795+0000: [CMS-concurrent-preclean-start] 2015-08-18T17:41:52.799+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-18T17:41:52.832+0000: [CMS-concurrent-sweep-start] 2015-08-18T17:41:53.046+0000: [CMS-concurrent-sweep: 0.214/0.214 secs] [Times: user=0.24 sys=0.00, real=0.21 secs] 2015-08-18T17:41:53.046+0000: [CMS-concurrent-reset-start] 2015-08-18T17:41:53.066+0000: [CMS-concurrent-reset: 0.020/0.020 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-18 17:42:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.61 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=7, accesses=857372, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=209, evicted=20, evictedPerRun=0.09569378197193146 2015-08-18 17:45:47,511 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919658729 with entries=49251, filesize=121.84 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919947142 2015-08-18 17:45:52,694 INFO [sync.0] wal.FSHLog: Slow sync cost: 192 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:45:52,694 INFO [sync.4] wal.FSHLog: Slow sync cost: 206 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:45:52,694 INFO [sync.2] wal.FSHLog: Slow sync cost: 125 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:45:52,695 INFO [sync.3] wal.FSHLog: Slow sync cost: 232 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:45:52,695 INFO [sync.1] wal.FSHLog: Slow sync cost: 184 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:46:44,925 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:46:45,741 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69027711, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/73ea70ead3924e858f99f6d697ac9e11 2015-08-18 17:46:45,786 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/73ea70ead3924e858f99f6d697ac9e11, entries=364421, sequenceid=69027711, filesize=8.8 M 2015-08-18 17:46:45,788 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218952, currentsize=150.80 KB/154424 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 863ms, sequenceid=69027711, compaction requested=true 2015-08-18 17:46:45,790 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 17:46:45,791 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=26.5 M 2015-08-18 17:46:45,797 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=7, currentSize=4834632, freeSize=4284785080, maxSize=4289619712, heapSize=4834632, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:46:48,266 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into ca28212a566449a9a22ece233fbcf9ac(size=26.2 M), total size for store is 4.5 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-18 17:46:48,267 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=26.5 M, priority=21, time=3457134201857443; duration=2sec 2015-08-18 17:47:35,347 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:47:36,130 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67019275, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/d742dcdefc864bafa15d36b758fa1105 2015-08-18 17:47:36,175 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/d742dcdefc864bafa15d36b758fa1105, entries=348225, sequenceid=67019275, filesize=8.7 M 2015-08-18 17:47:36,176 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222344, currentsize=105.90 KB/108440 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 829ms, sequenceid=67019275, compaction requested=true 2015-08-18 17:47:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.61 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=7, accesses=861435, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=239, evicted=20, evictedPerRun=0.08368200808763504 2015-08-18 17:49:59,833 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:49:59,997 INFO [Thread-342] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 17:50:00,015 INFO [Thread-342] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789767_49257 2015-08-18 17:50:00,069 INFO [Thread-342] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 17:50:00,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67621453, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/b450bb3bca6743cf916dc3f24270adb0 2015-08-18 17:50:00,765 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/b450bb3bca6743cf916dc3f24270adb0, entries=387056, sequenceid=67621453, filesize=9.0 M 2015-08-18 17:50:00,766 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220240, currentsize=108.05 KB/110648 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 934ms, sequenceid=67621453, compaction requested=true 2015-08-18 17:50:00,768 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 17:50:00,768 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=228.1 M 2015-08-18 17:50:00,775 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=7, currentSize=4834632, freeSize=4284785080, maxSize=4289619712, heapSize=4834632, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 17:50:19,388 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 00448d3a02e84c57a165cb4a3632adf4(size=228.3 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 18sec to execute. 2015-08-18 17:50:19,388 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=228.1 M, priority=21, time=3457329179478790; duration=18sec 2015-08-18 17:50:31,830 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.15:10110] 2015-08-18 17:50:31,890 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919947142 with entries=48924, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920231646 2015-08-18 17:50:31,891 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919086213 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919086213 2015-08-18 17:52:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.60 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=6, accesses=895163, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=269, evicted=21, evictedPerRun=0.07806691527366638 2015-08-18 17:54:03,731 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:54:04,543 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=60946297, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/02ab21ff964f49e2863e95ed96c6a0a3 2015-08-18 17:54:04,631 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/02ab21ff964f49e2863e95ed96c6a0a3, entries=367721, sequenceid=60946297, filesize=8.8 M 2015-08-18 17:54:04,632 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219320, currentsize=82 KB/83968 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 900ms, sequenceid=60946297, compaction requested=true 2015-08-18 17:55:17,002 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920231646 with entries=48014, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920516747 2015-08-18 17:55:17,004 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919373384 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919373384 2015-08-18 17:55:17,102 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919658729 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919658729 2015-08-18T17:55:17.158+0000: [CMS-concurrent-mark-start] 2015-08-18T17:55:17.215+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.28 sys=0.01, real=0.06 secs] 2015-08-18T17:55:17.215+0000: [CMS-concurrent-preclean-start] 2015-08-18T17:55:17.219+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-18T17:55:17.260+0000: [CMS-concurrent-sweep-start] 2015-08-18T17:55:17.497+0000: [CMS-concurrent-sweep: 0.237/0.237 secs] [Times: user=0.25 sys=0.00, real=0.24 secs] 2015-08-18T17:55:17.497+0000: [CMS-concurrent-reset-start] 2015-08-18T17:55:17.526+0000: [CMS-concurrent-reset: 0.030/0.030 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 2015-08-18 17:57:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.60 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=6, accesses=895163, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=299, evicted=21, evictedPerRun=0.07023411244153976 2015-08-18 17:59:18,170 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 17:59:18,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69093377, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/3fab145b70294a9ea5ddcb68d86d3b7f 2015-08-18 17:59:19,029 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/3fab145b70294a9ea5ddcb68d86d3b7f, entries=362108, sequenceid=69093377, filesize=8.8 M 2015-08-18 17:59:19,031 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220272, currentsize=91.69 KB/93888 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 862ms, sequenceid=69093377, compaction requested=true 2015-08-18 18:00:01,431 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 149 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.1:10110] 2015-08-18 18:00:01,490 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920516747 with entries=47608, filesize=121.77 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920801207 2015-08-18 18:00:15,326 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:00:16,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67083962, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/db15f89abd7943b29246476aee3ff694 2015-08-18 18:00:16,169 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/db15f89abd7943b29246476aee3ff694, entries=348160, sequenceid=67083962, filesize=8.7 M 2015-08-18 18:00:16,171 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218368, currentsize=106.84 KB/109400 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 844ms, sequenceid=67083962, compaction requested=true 2015-08-18 18:00:16,173 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 18:00:16,173 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp, totalSize=26.1 M 2015-08-18 18:00:16,180 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=6, currentSize=4825792, freeSize=4284793920, maxSize=4289619712, heapSize=4825792, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:00:18,774 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into f85ece195c9f4ffba4b5f43357933be7(size=25.8 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-18 18:00:18,775 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., storeName=e, fileCount=3, fileSize=26.1 M, priority=20, time=3457944584409952; duration=2sec 2015-08-18 18:02:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.60 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=6, accesses=899316, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=329, evicted=21, evictedPerRun=0.06382978707551956 2015-08-18 18:03:43,247 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:03:44,055 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67692523, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/ef725e9e452b40b8b9de2776101d56c1 2015-08-18 18:03:44,093 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/ef725e9e452b40b8b9de2776101d56c1, entries=386939, sequenceid=67692523, filesize=9.0 M 2015-08-18 18:03:44,094 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219920, currentsize=142.50 KB/145920 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 847ms, sequenceid=67692523, compaction requested=true 2015-08-18 18:04:47,671 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920801207 with entries=49019, filesize=121.68 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921087466 2015-08-18 18:04:47,672 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919947142 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439919947142 2015-08-18 18:07:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.60 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=6, accesses=899316, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=359, evicted=21, evictedPerRun=0.05849582329392433 2015-08-18T18:08:53.249+0000: [CMS-concurrent-mark-start] 2015-08-18T18:08:53.331+0000: [CMS-concurrent-mark: 0.082/0.082 secs] [Times: user=0.42 sys=0.00, real=0.08 secs] 2015-08-18T18:08:53.331+0000: [CMS-concurrent-preclean-start] 2015-08-18T18:08:53.336+0000: [CMS-concurrent-preclean: 0.004/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-18T18:08:53.336+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18T18:08:55.879+0000: [CMS-concurrent-abortable-preclean: 0.540/2.543 secs] [Times: user=0.74 sys=0.07, real=2.54 secs] 2015-08-18T18:08:55.934+0000: [CMS-concurrent-sweep-start] 2015-08-18T18:08:56.134+0000: [CMS-concurrent-sweep: 0.200/0.200 secs] [Times: user=0.22 sys=0.01, real=0.20 secs] 2015-08-18T18:08:56.134+0000: [CMS-concurrent-reset-start] 2015-08-18T18:08:56.163+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 2015-08-18 18:09:03,227 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:09:03,350 INFO [Thread-404] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 18:09:03,351 INFO [Thread-404] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789866_49356 2015-08-18 18:09:03,390 INFO [Thread-404] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 18:09:04,015 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61018498, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/7c1edea735c943f38b98f1a7b5c73525 2015-08-18 18:09:04,061 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/7c1edea735c943f38b98f1a7b5c73525, entries=368261, sequenceid=61018498, filesize=8.8 M 2015-08-18 18:09:04,062 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221416, currentsize=115.48 KB/118256 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 835ms, sequenceid=61018498, compaction requested=true 2015-08-18 18:09:04,066 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 18:09:04,066 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 5 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=387.0 M 2015-08-18 18:09:04,077 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=6, currentSize=4825792, freeSize=4284793920, maxSize=4289619712, heapSize=4825792, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:09:22,482 WARN [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hdfs.BlockReaderFactory: I/O error constructing remote block reader. java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884) at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:747) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:662) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:326) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:793) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:678) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1372) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1591) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1470) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:437) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:708) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1251) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:139) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:596) at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:255) at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:112) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1212) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1798) at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:519) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2015-08-18 18:09:22,483 WARN [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hdfs.DFSClient: Failed to connect to /172.24.32.8:10110 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884) at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:747) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:662) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:326) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:793) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:678) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1372) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1591) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1470) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:437) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:708) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1251) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:139) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:596) at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:255) at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:112) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1212) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1798) at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:519) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2015-08-18 18:09:22,485 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hdfs.DFSClient: Successfully connected to /172.24.32.9:10110 for BP-1466258523-172.24.32.1-1437768622582:blk_1073787045_46523 2015-08-18 18:09:31,402 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921087466 with entries=49661, filesize=121.68 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921371226 2015-08-18 18:09:31,405 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920231646 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920231646 2015-08-18 18:09:34,135 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 5 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 75f39f9e14494f79beaad3f71b8f93f9(size=387.4 M), total size for store is 3.1 G. This selection was in queue for 0sec, and took 30sec to execute. 2015-08-18 18:09:34,135 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=5, fileSize=387.0 M, priority=18, time=3458472477319191; duration=30sec 2015-08-18 18:11:55,828 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:11:55,952 INFO [Thread-418] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 18:11:55,953 INFO [Thread-418] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789885_49375 2015-08-18 18:11:55,983 INFO [Thread-418] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 18:11:56,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69160424, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/2e92a0b0fe3d4d12a8921bbeb0620fad 2015-08-18 18:11:56,679 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/2e92a0b0fe3d4d12a8921bbeb0620fad, entries=363520, sequenceid=69160424, filesize=8.8 M 2015-08-18 18:11:56,680 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219848, currentsize=113.81 KB/116544 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 852ms, sequenceid=69160424, compaction requested=true 2015-08-18 18:11:56,683 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 18:11:56,683 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=43.9 M 2015-08-18 18:11:56,688 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:11:56,752 INFO [Thread-420] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 18:11:56,752 INFO [Thread-420] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789887_49377 2015-08-18 18:11:56,784 INFO [Thread-420] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 18:12:00,359 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 362084f89cf041fba17c8fbe5ff34149(size=43.7 M), total size for store is 4.5 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-18 18:12:00,360 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=43.9 M, priority=21, time=3458645094408723; duration=3sec 2015-08-18 18:12:32,141 INFO [sync.4] wal.FSHLog: Slow sync cost: 128 ms, current pipeline: [172.24.32.16:10110, 172.24.32.6:10110, 172.24.32.11:10110] 2015-08-18 18:12:32,141 INFO [sync.0] wal.FSHLog: Slow sync cost: 112 ms, current pipeline: [172.24.32.16:10110, 172.24.32.6:10110, 172.24.32.11:10110] 2015-08-18 18:12:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=965728, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=389, evicted=23, evictedPerRun=0.05912596359848976 2015-08-18 18:12:55,210 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:12:56,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67149948, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/2af10f55294d4312b8c1a2c8cfcc832e 2015-08-18 18:12:56,260 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/2af10f55294d4312b8c1a2c8cfcc832e, entries=348084, sequenceid=67149948, filesize=8.7 M 2015-08-18 18:12:56,261 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219864, currentsize=292.37 KB/299384 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 1051ms, sequenceid=67149948, compaction requested=true 2015-08-18 18:14:16,107 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921371226 with entries=48765, filesize=121.77 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921655672 2015-08-18 18:14:16,109 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920516747 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920516747 2015-08-18 18:17:25,613 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:17:26,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67764890, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/d6a2229e396c46c794a87e4946247608 2015-08-18 18:17:26,504 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/d6a2229e396c46c794a87e4946247608, entries=386967, sequenceid=67764890, filesize=9.0 M 2015-08-18 18:17:26,505 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221720, currentsize=180.86 KB/185200 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 893ms, sequenceid=67764890, compaction requested=true 2015-08-18 18:17:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=965728, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=419, evicted=23, evictedPerRun=0.05489259958267212 2015-08-18 18:19:00,685 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.7:10110, 172.24.32.1:10110] 2015-08-18 18:19:00,744 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921655672 with entries=48154, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921940507 2015-08-18 18:19:00,746 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920801207 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439920801207 2015-08-18T18:19:06.983+0000: [CMS-concurrent-mark-start] 2015-08-18T18:19:07.047+0000: [CMS-concurrent-mark: 0.064/0.064 secs] [Times: user=0.32 sys=0.00, real=0.06 secs] 2015-08-18T18:19:07.047+0000: [CMS-concurrent-preclean-start] 2015-08-18T18:19:07.051+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2015-08-18T18:19:07.051+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18T18:19:10.195+0000: [CMS-concurrent-abortable-preclean: 0.940/3.144 secs] [Times: user=1.18 sys=0.05, real=3.14 secs] 2015-08-18T18:19:10.233+0000: [CMS-concurrent-sweep-start] 2015-08-18T18:19:10.451+0000: [CMS-concurrent-sweep: 0.218/0.218 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-18T18:19:10.452+0000: [CMS-concurrent-reset-start] 2015-08-18T18:19:10.480+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-18 18:22:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=965728, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=449, evicted=23, evictedPerRun=0.05122494325041771 2015-08-18 18:23:50,232 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.3:10110, 172.24.32.15:10110] 2015-08-18 18:23:50,292 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921940507 with entries=48728, filesize=121.72 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922230050 2015-08-18 18:24:09,312 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:24:10,101 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61090708, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/76e16b74ab894e31b06e03e4a92ff683 2015-08-18 18:24:10,164 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/76e16b74ab894e31b06e03e4a92ff683, entries=367934, sequenceid=61090708, filesize=8.8 M 2015-08-18 18:24:10,165 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221968, currentsize=132.78 KB/135968 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 853ms, sequenceid=61090708, compaction requested=true 2015-08-18 18:24:35,412 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:24:35,632 INFO [Thread-463] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 18:24:35,632 INFO [Thread-463] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789933_49424 2015-08-18 18:24:35,687 INFO [Thread-463] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 18:24:36,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69226786, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/9f982de6216f4eefbac30905395eb847 2015-08-18 18:24:36,316 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/9f982de6216f4eefbac30905395eb847, entries=363214, sequenceid=69226786, filesize=8.8 M 2015-08-18 18:24:36,317 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134228312, currentsize=141.88 KB/145280 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 905ms, sequenceid=69226786, compaction requested=true 2015-08-18 18:25:01,364 INFO [sync.2] wal.FSHLog: Slow sync cost: 130 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.7:10110] 2015-08-18 18:25:39,069 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:25:39,852 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67214265, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/b5b9b0a9af63467ca07f15fd7d6e0699 2015-08-18 18:25:39,898 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/b5b9b0a9af63467ca07f15fd7d6e0699, entries=347856, sequenceid=67214265, filesize=8.7 M 2015-08-18 18:25:39,899 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218728, currentsize=129.06 KB/132160 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 831ms, sequenceid=67214265, compaction requested=true 2015-08-18 18:25:39,901 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 18:25:39,902 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp, totalSize=43.2 M 2015-08-18 18:25:39,909 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:25:43,561 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into 28d803b33099408da06555bf0dc6d0c2(size=43.0 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-18 18:25:43,561 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., storeName=e, fileCount=3, fileSize=43.2 M, priority=20, time=3459468312842830; duration=3sec 2015-08-18 18:27:09,478 INFO [sync.0] wal.FSHLog: Slow sync cost: 113 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.7:10110] 2015-08-18 18:27:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=972614, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=479, evicted=23, evictedPerRun=0.04801670089364052 2015-08-18 18:28:34,311 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.7:10110] 2015-08-18 18:28:34,362 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922230050 with entries=48020, filesize=121.80 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922513706 2015-08-18 18:28:34,364 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921087466 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921087466 2015-08-18 18:28:34,453 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921371226 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921371226 2015-08-18 18:31:11,833 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:31:11,985 INFO [Thread-484] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 18:31:11,986 INFO [Thread-484] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789952_49443 2015-08-18 18:31:12,032 INFO [Thread-484] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 18:31:12,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67836985, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/77558b8791924c4fa2a46bdf6f0e8b04 2015-08-18 18:31:12,728 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/77558b8791924c4fa2a46bdf6f0e8b04, entries=386633, sequenceid=67836985, filesize=9.0 M 2015-08-18 18:31:12,730 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220520, currentsize=121.84 KB/124760 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 897ms, sequenceid=67836985, compaction requested=true 2015-08-18 18:31:12,733 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 18:31:12,733 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=26.9 M 2015-08-18 18:31:12,738 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:31:15,357 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 1790520684d643f5a7f5e26b33bc8ec7(size=26.7 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-18 18:31:15,358 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=26.9 M, priority=20, time=3459801144146478; duration=2sec 2015-08-18T18:32:24.074+0000: [CMS-concurrent-mark-start] 2015-08-18T18:32:24.131+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.29 sys=0.00, real=0.05 secs] 2015-08-18T18:32:24.131+0000: [CMS-concurrent-preclean-start] 2015-08-18T18:32:24.135+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-18T18:32:24.171+0000: [CMS-concurrent-sweep-start] 2015-08-18T18:32:24.393+0000: [CMS-concurrent-sweep: 0.222/0.222 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-18T18:32:24.393+0000: [CMS-concurrent-reset-start] 2015-08-18T18:32:24.423+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 2015-08-18 18:32:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=976548, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=509, evicted=23, evictedPerRun=0.045186638832092285 2015-08-18 18:33:20,721 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922513706 with entries=49481, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922800495 2015-08-18 18:33:20,723 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921655672 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921655672 2015-08-18 18:33:20,821 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921940507 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439921940507 2015-08-18 18:37:12,671 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:37:12,849 INFO [Thread-502] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526) 2015-08-18 18:37:12,849 INFO [Thread-502] hdfs.DFSClient: Abandoning BP-1466258523-172.24.32.1-1437768622582:blk_1073789974_49465 2015-08-18 18:37:12,894 INFO [Thread-502] hdfs.DFSClient: Excluding datanode 172.24.32.8:10110 2015-08-18 18:37:13,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69294353, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/259195161bd64f49b827c9e539c76799 2015-08-18 18:37:13,591 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/259195161bd64f49b827c9e539c76799, entries=362603, sequenceid=69294353, filesize=8.8 M 2015-08-18 18:37:13,592 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218472, currentsize=155.01 KB/158728 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 921ms, sequenceid=69294353, compaction requested=true 2015-08-18 18:37:13,595 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 18:37:13,595 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=61.3 M 2015-08-18 18:37:13,600 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:37:18,837 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 6c4f87880cd249ae9095015fc4e2460a(size=61.1 M), total size for store is 4.5 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-18 18:37:18,837 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=61.3 M, priority=21, time=3460162006213437; duration=5sec 2015-08-18 18:37:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=985979, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=539, evicted=23, evictedPerRun=0.04267161339521408 2015-08-18 18:38:04,517 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.3:10110, 172.24.32.7:10110] 2015-08-18 18:38:04,568 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922800495 with entries=50079, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923084309 2015-08-18 18:38:21,032 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:38:21,835 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67280533, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/eafd01142370415c845dfa0bcfbd219d 2015-08-18 18:38:21,898 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/eafd01142370415c845dfa0bcfbd219d, entries=347926, sequenceid=67280533, filesize=8.7 M 2015-08-18 18:38:21,899 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218280, currentsize=147.90 KB/151448 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 867ms, sequenceid=67280533, compaction requested=true 2015-08-18 18:39:12,251 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.06 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:39:13,074 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61164267, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/c0465993d7554d8cb65ff0650d7b5649 2015-08-18 18:39:13,137 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/c0465993d7554d8cb65ff0650d7b5649, entries=368276, sequenceid=61164267, filesize=8.8 M 2015-08-18 18:39:13,138 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134285216, currentsize=84.81 KB/86848 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 887ms, sequenceid=61164267, compaction requested=true 2015-08-18 18:42:27,519 INFO [main-EventThread] replication.ReplicationTrackerZKImpl: /hbase/rs/r12s15.sjc.aristanetworks.com,9104,1439917658834 znode expired, triggering replicatorRemoved event 2015-08-18 18:42:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=985979, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=569, evicted=23, evictedPerRun=0.04042179137468338 2015-08-18 18:43:09,008 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923084309 with entries=43121, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923388915 2015-08-18 18:43:09,010 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922230050 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922230050 2015-08-18 18:43:20,219 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Atomically moving r12s15.sjc.aristanetworks.com,9104,1439917658834's wals to my queue 2015-08-18 18:44:49,597 INFO [sync.2] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.11:10110] 2015-08-18 18:45:16,990 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:45:17,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67907143, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/6963eac458d9439c9622e8375a23b80f 2015-08-18 18:45:17,824 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/6963eac458d9439c9622e8375a23b80f, entries=386715, sequenceid=67907143, filesize=9.0 M 2015-08-18 18:45:17,825 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134227168, currentsize=145.87 KB/149368 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 835ms, sequenceid=67907143, compaction requested=true 2015-08-18T18:46:10.046+0000: [CMS-concurrent-mark-start] 2015-08-18T18:46:10.100+0000: [CMS-concurrent-mark: 0.054/0.054 secs] [Times: user=0.27 sys=0.00, real=0.05 secs] 2015-08-18T18:46:10.100+0000: [CMS-concurrent-preclean-start] 2015-08-18T18:46:10.104+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2015-08-18T18:46:10.104+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18T18:46:13.273+0000: [CMS-concurrent-abortable-preclean: 0.553/3.169 secs] [Times: user=0.79 sys=0.05, real=3.17 secs] 2015-08-18T18:46:13.309+0000: [CMS-concurrent-sweep-start] 2015-08-18T18:46:13.526+0000: [CMS-concurrent-sweep: 0.218/0.218 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-18T18:46:13.527+0000: [CMS-concurrent-reset-start] 2015-08-18T18:46:13.555+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 2015-08-18 18:46:25,579 INFO [sync.2] wal.FSHLog: Slow sync cost: 112 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.11:10110] 2015-08-18 18:47:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=985979, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=599, evicted=23, evictedPerRun=0.038397327065467834 2015-08-18 18:47:56,329 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923388915 with entries=48736, filesize=121.65 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923676130 2015-08-18 18:47:56,330 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922513706 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922513706 2015-08-18 18:50:09,197 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:50:10,218 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69357629, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/242c04ad8a7f442dbdead63622436e1c 2015-08-18 18:50:10,280 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/242c04ad8a7f442dbdead63622436e1c, entries=362956, sequenceid=69357629, filesize=8.8 M 2015-08-18 18:50:10,281 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220736, currentsize=175.20 KB/179400 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 1084ms, sequenceid=69357629, compaction requested=true 2015-08-18 18:51:17,729 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:51:18,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67342598, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/ec7ebb6bb11d45349d9ee93fe85a4ce4 2015-08-18 18:51:18,613 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/ec7ebb6bb11d45349d9ee93fe85a4ce4, entries=345981, sequenceid=67342598, filesize=8.7 M 2015-08-18 18:51:18,614 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218352, currentsize=239.51 KB/245256 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 885ms, sequenceid=67342598, compaction requested=true 2015-08-18 18:51:18,617 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 18:51:18,617 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp, totalSize=60.3 M 2015-08-18 18:51:18,622 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:51:23,734 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into 5aebd3ffd32e444193a08511bd320657(size=60.1 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-18 18:51:23,734 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., storeName=e, fileCount=3, fileSize=60.3 M, priority=20, time=3461007028262741; duration=5sec 2015-08-18 18:52:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=995613, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=629, evicted=23, evictedPerRun=0.036565978080034256 2015-08-18 18:52:41,268 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923676130 with entries=47746, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923961171 2015-08-18 18:52:41,269 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922800495 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439922800495 2015-08-18 18:54:35,606 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:54:36,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61233600, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/1a80162c242940ddad690139d6bb0934 2015-08-18 18:54:36,391 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/1a80162c242940ddad690139d6bb0934, entries=367616, sequenceid=61233600, filesize=8.8 M 2015-08-18 18:54:36,392 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221128, currentsize=66.55 KB/68144 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 786ms, sequenceid=61233600, compaction requested=true 2015-08-18 18:54:36,395 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 18:54:36,395 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=26.5 M 2015-08-18 18:54:36,401 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:54:38,713 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 417460ed0bba473987cdc3d821be18f2(size=26.2 M), total size for store is 3.1 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-18 18:54:38,713 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=26.5 M, priority=19, time=3461204806269984; duration=2sec 2015-08-18 18:57:25,549 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923961171 with entries=48582, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924245371 2015-08-18 18:57:25,553 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923084309 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923084309 2015-08-18 18:57:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=999654, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=659, evicted=23, evictedPerRun=0.034901365637779236 2015-08-18 18:59:01,154 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 18:59:02,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67978810, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/f443dd307328460c867d23ef3313112b 2015-08-18 18:59:02,220 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/f443dd307328460c867d23ef3313112b, entries=386614, sequenceid=67978810, filesize=9.0 M 2015-08-18 18:59:02,221 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134226528, currentsize=183.50 KB/187904 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 1067ms, sequenceid=67978810, compaction requested=true 2015-08-18 18:59:02,223 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 18:59:02,224 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=44.7 M 2015-08-18 18:59:02,229 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 18:59:05,967 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 0b785ac486434c3a8b884ba8230ef207(size=44.5 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-18 18:59:05,968 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=44.7 M, priority=20, time=3461470635031072; duration=3sec 2015-08-18T18:59:32.765+0000: [CMS-concurrent-mark-start] 2015-08-18T18:59:32.819+0000: [CMS-concurrent-mark: 0.053/0.053 secs] [Times: user=0.27 sys=0.00, real=0.05 secs] 2015-08-18T18:59:32.819+0000: [CMS-concurrent-preclean-start] 2015-08-18T18:59:32.823+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-18T18:59:32.859+0000: [CMS-concurrent-sweep-start] 2015-08-18T18:59:33.082+0000: [CMS-concurrent-sweep: 0.222/0.222 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-18T18:59:33.082+0000: [CMS-concurrent-reset-start] 2015-08-18T18:59:33.111+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-18 19:02:10,663 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924245371 with entries=48424, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924530551 2015-08-18 19:02:10,664 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923388915 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923388915 2015-08-18 19:02:15,867 INFO [sync.4] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:02:15,867 INFO [sync.2] wal.FSHLog: Slow sync cost: 202 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:02:15,867 INFO [sync.0] wal.FSHLog: Slow sync cost: 157 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:02:15,867 INFO [sync.1] wal.FSHLog: Slow sync cost: 222 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:02:15,867 INFO [sync.3] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:02:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1006176, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=689, evicted=23, evictedPerRun=0.033381711691617966 2015-08-18 19:02:53,231 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:02:54,000 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69424088, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/673672d442d64693b25aa5f005d0778d 2015-08-18 19:02:54,037 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/673672d442d64693b25aa5f005d0778d, entries=364956, sequenceid=69424088, filesize=8.8 M 2015-08-18 19:02:54,038 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221240, currentsize=96.98 KB/99312 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 807ms, sequenceid=69424088, compaction requested=true 2015-08-18 19:02:54,040 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 19:02:54,040 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=78.8 M 2015-08-18 19:02:54,050 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:03:00,401 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 3b6d388b53084c41a29f0546478d6777(size=78.6 M), total size for store is 4.5 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-18 19:03:00,401 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=78.8 M, priority=21, time=3461702451328889; duration=6sec 2015-08-18 19:03:59,408 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:04:00,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67408028, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/ed77d5837a434de9ab992f3cd480d0c7 2015-08-18 19:04:00,244 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/ed77d5837a434de9ab992f3cd480d0c7, entries=347283, sequenceid=67408028, filesize=8.7 M 2015-08-18 19:04:00,245 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220600, currentsize=290.94 KB/297920 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 837ms, sequenceid=67408028, compaction requested=true 2015-08-18 19:05:04,652 INFO [sync.1] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:06:25,567 INFO [sync.0] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.5:10110] 2015-08-18 19:06:57,568 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924530551 with entries=48546, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924817414 2015-08-18 19:06:57,570 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923676130 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923676130 2015-08-18 19:07:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1018291, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=719, evicted=23, evictedPerRun=0.031988874077796936 2015-08-18 19:09:41,232 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:09:41,977 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61305208, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/70f921f491e0441ebb4966178f0b6897 2015-08-18 19:09:42,047 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/70f921f491e0441ebb4966178f0b6897, entries=368298, sequenceid=61305208, filesize=8.8 M 2015-08-18 19:09:42,048 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223360, currentsize=222.18 KB/227512 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 816ms, sequenceid=61305208, compaction requested=true 2015-08-18 19:11:43,280 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924817414 with entries=47787, filesize=121.71 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925103116 2015-08-18 19:11:43,282 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923961171 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439923961171 2015-08-18 19:11:48,456 INFO [sync.2] wal.FSHLog: Slow sync cost: 206 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:11:48,456 INFO [sync.3] wal.FSHLog: Slow sync cost: 187 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:11:48,457 INFO [sync.4] wal.FSHLog: Slow sync cost: 151 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:11:48,456 INFO [sync.1] wal.FSHLog: Slow sync cost: 210 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:11:48,457 INFO [sync.0] wal.FSHLog: Slow sync cost: 147 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:12:01,810 INFO [sync.4] wal.FSHLog: Slow sync cost: 118 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:12:01,810 INFO [sync.0] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:12:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1018291, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=749, evicted=23, evictedPerRun=0.03070761077105999 2015-08-18 19:12:46,767 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:12:47,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=68050180, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/022087e98ed340af8de34fcbd2a56bf1 2015-08-18 19:12:47,631 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/022087e98ed340af8de34fcbd2a56bf1, entries=386747, sequenceid=68050180, filesize=9.0 M 2015-08-18 19:12:47,632 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220520, currentsize=175.79 KB/180008 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 865ms, sequenceid=68050180, compaction requested=true 2015-08-18T19:13:04.402+0000: [CMS-concurrent-mark-start] 2015-08-18T19:13:04.456+0000: [CMS-concurrent-mark: 0.054/0.054 secs] [Times: user=0.28 sys=0.00, real=0.05 secs] 2015-08-18T19:13:04.456+0000: [CMS-concurrent-preclean-start] 2015-08-18T19:13:04.460+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-18T19:13:04.460+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18T19:13:07.399+0000: [CMS-concurrent-abortable-preclean: 0.935/2.939 secs] [Times: user=1.15 sys=0.06, real=2.94 secs] 2015-08-18T19:13:07.462+0000: [CMS-concurrent-sweep-start] 2015-08-18T19:13:07.677+0000: [CMS-concurrent-sweep: 0.216/0.216 secs] [Times: user=0.24 sys=0.00, real=0.21 secs] 2015-08-18T19:13:07.678+0000: [CMS-concurrent-reset-start] 2015-08-18T19:13:07.707+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 2015-08-18 19:13:37,466 INFO [sync.2] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:15:28,812 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:15:29,587 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69489184, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/f7aedf93507a400c8dc67bb1e7d1f9d7 2015-08-18 19:15:29,632 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/f7aedf93507a400c8dc67bb1e7d1f9d7, entries=362922, sequenceid=69489184, filesize=8.8 M 2015-08-18 19:15:29,633 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217864, currentsize=136.61 KB/139888 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 821ms, sequenceid=69489184, compaction requested=true 2015-08-18 19:15:32,697 INFO [sync.3] wal.FSHLog: Slow sync cost: 265 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:15:32,697 INFO [sync.2] wal.FSHLog: Slow sync cost: 327 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:15:32,697 INFO [sync.4] wal.FSHLog: Slow sync cost: 252 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:15:32,697 INFO [sync.1] wal.FSHLog: Slow sync cost: 332 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:15:32,697 INFO [sync.0] wal.FSHLog: Slow sync cost: 192 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.8:10110] 2015-08-18 19:16:28,310 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925103116 with entries=47158, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925388128 2015-08-18 19:16:28,311 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924245371 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924245371 2015-08-18 19:16:40,054 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:16:40,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67471845, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/f058416e571e40aa90736d53ece2e37c 2015-08-18 19:16:40,873 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/f058416e571e40aa90736d53ece2e37c, entries=348225, sequenceid=67471845, filesize=8.7 M 2015-08-18 19:16:40,874 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222752, currentsize=105.61 KB/108144 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 820ms, sequenceid=67471845, compaction requested=true 2015-08-18 19:16:40,877 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 19:16:40,877 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp, totalSize=77.5 M 2015-08-18 19:16:40,883 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:16:47,153 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into 7299f7977544469cbc63b74b659981c1(size=77.3 M), total size for store is 4.9 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-18 19:16:47,153 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., storeName=e, fileCount=3, fileSize=77.5 M, priority=20, time=3462529288369167; duration=6sec 2015-08-18 19:17:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1030672, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=779, evicted=23, evictedPerRun=0.02952503226697445 2015-08-18 19:21:14,038 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925388128 with entries=47368, filesize=121.73 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925673668 2015-08-18 19:21:14,041 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924530551 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924530551 2015-08-18 19:22:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1030672, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=809, evicted=23, evictedPerRun=0.028430160135030746 2015-08-18 19:24:47,452 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:24:48,178 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61376453, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/911258c904ed4342ad288d5bd0067ffe 2015-08-18 19:24:48,206 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/911258c904ed4342ad288d5bd0067ffe, entries=367855, sequenceid=61376453, filesize=8.8 M 2015-08-18 19:24:48,207 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218320, currentsize=96.70 KB/99016 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 754ms, sequenceid=61376453, compaction requested=true 2015-08-18 19:24:48,209 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-18 19:24:48,210 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=43.8 M 2015-08-18 19:24:48,217 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:24:52,003 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 5449bb9966bf48aa853aa3b8d9a87cff(size=43.6 M), total size for store is 3.1 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-18 19:24:52,004 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=43.8 M, priority=19, time=3463016620798032; duration=3sec 2015-08-18 19:26:02,162 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925673668 with entries=49543, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925962072 2015-08-18 19:26:02,163 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924817414 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439924817414 2015-08-18 19:26:32,949 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18T19:26:33.370+0000: [CMS-concurrent-mark-start] 2015-08-18T19:26:33.449+0000: [CMS-concurrent-mark: 0.079/0.079 secs] [Times: user=0.48 sys=0.00, real=0.08 secs] 2015-08-18T19:26:33.449+0000: [CMS-concurrent-preclean-start] 2015-08-18T19:26:33.454+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2015-08-18T19:26:33.454+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18 19:26:33,748 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=68120699, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/620ae0c81a0d438ca34e9e7b3f441b2e 2015-08-18 19:26:33,767 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/620ae0c81a0d438ca34e9e7b3f441b2e, entries=386566, sequenceid=68120699, filesize=9.0 M 2015-08-18 19:26:33,768 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219624, currentsize=198.73 KB/203496 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 819ms, sequenceid=68120699, compaction requested=true 2015-08-18 19:26:33,771 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 19:26:33,771 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=62.5 M 2015-08-18 19:26:33,776 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18T19:26:34.005+0000: [CMS-concurrent-abortable-preclean: 0.155/0.551 secs] [Times: user=0.80 sys=0.02, real=0.55 secs] 2015-08-18T19:26:34.044+0000: [CMS-concurrent-sweep-start] 2015-08-18T19:26:34.263+0000: [CMS-concurrent-sweep: 0.205/0.219 secs] [Times: user=0.56 sys=0.00, real=0.22 secs] 2015-08-18T19:26:34.263+0000: [CMS-concurrent-reset-start] 2015-08-18T19:26:34.293+0000: [CMS-concurrent-reset: 0.030/0.030 secs] [Times: user=0.06 sys=0.00, real=0.03 secs] 2015-08-18 19:26:38,698 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 689a4ec9be6b4237a24cc6893a96fd1d(size=62.3 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-18 19:26:38,698 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=62.5 M, priority=20, time=3463122182298192; duration=4sec 2015-08-18 19:26:51,850 INFO [sync.1] wal.FSHLog: Slow sync cost: 205 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:26:51,850 INFO [sync.0] wal.FSHLog: Slow sync cost: 224 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:26:51,851 INFO [sync.4] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:26:51,851 INFO [sync.3] wal.FSHLog: Slow sync cost: 181 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:26:51,852 INFO [sync.2] wal.FSHLog: Slow sync cost: 202 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1046498, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=839, evicted=23, evictedPerRun=0.027413588017225266 2015-08-18 19:27:52,891 INFO [sync.3] wal.FSHLog: Slow sync cost: 201 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:52,891 INFO [sync.4] wal.FSHLog: Slow sync cost: 182 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:52,891 INFO [sync.1] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:52,891 INFO [sync.0] wal.FSHLog: Slow sync cost: 126 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:56,234 INFO [sync.0] wal.FSHLog: Slow sync cost: 928 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:56,234 INFO [sync.1] wal.FSHLog: Slow sync cost: 907 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:56,234 INFO [sync.4] wal.FSHLog: Slow sync cost: 964 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:56,234 INFO [sync.3] wal.FSHLog: Slow sync cost: 985 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:56,234 INFO [sync.2] wal.FSHLog: Slow sync cost: 884 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:59,142 INFO [sync.4] wal.FSHLog: Slow sync cost: 597 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:59,150 INFO [sync.3] wal.FSHLog: Slow sync cost: 525 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:59,150 INFO [sync.2] wal.FSHLog: Slow sync cost: 541 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:59,150 INFO [sync.0] wal.FSHLog: Slow sync cost: 560 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:27:59,150 INFO [sync.1] wal.FSHLog: Slow sync cost: 545 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:28:04,515 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.09 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:28:05,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69555313, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/f265cfc8b71b4d7192927fdbde517514 2015-08-18 19:28:05,569 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/f265cfc8b71b4d7192927fdbde517514, entries=363357, sequenceid=69555313, filesize=8.8 M 2015-08-18 19:28:05,571 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.09 MB/134314656, currentsize=187.41 KB/191912 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 1057ms, sequenceid=69555313, compaction requested=true 2015-08-18 19:28:05,573 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 19:28:05,574 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=96.2 M 2015-08-18 19:28:05,579 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:28:13,083 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 4d0929d844bf4d4e859f8e881fa64836(size=96.0 M), total size for store is 4.5 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-18 19:28:13,083 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=96.2 M, priority=21, time=3463213984658898; duration=7sec 2015-08-18 19:29:20,454 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:29:21,349 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67536651, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/0da88cafbeae496ab82e0d6f1414b423 2015-08-18 19:29:21,385 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/0da88cafbeae496ab82e0d6f1414b423, entries=348288, sequenceid=67536651, filesize=8.7 M 2015-08-18 19:29:21,387 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223280, currentsize=241.51 KB/247304 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 933ms, sequenceid=67536651, compaction requested=true 2015-08-18 19:29:27,868 INFO [sync.2] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:29:53,483 INFO [sync.3] wal.FSHLog: Slow sync cost: 134 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:29:53,483 INFO [sync.2] wal.FSHLog: Slow sync cost: 174 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:29:53,484 INFO [sync.4] wal.FSHLog: Slow sync cost: 118 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:29:53,484 INFO [sync.0] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.11:10110] 2015-08-18 19:30:46,631 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925962072 with entries=48781, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926246435 2015-08-18 19:30:46,633 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925103116 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925103116 2015-08-18 19:30:46,689 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925388128 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925388128 2015-08-18 19:32:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1061302, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=869, evicted=23, evictedPerRun=0.026467204093933105 2015-08-18 19:35:31,869 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926246435 with entries=49607, filesize=121.92 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926531611 2015-08-18T19:36:55.072+0000: [CMS-concurrent-mark-start] 2015-08-18T19:36:55.161+0000: [CMS-concurrent-mark: 0.089/0.089 secs] [Times: user=0.46 sys=0.00, real=0.08 secs] 2015-08-18T19:36:55.161+0000: [CMS-concurrent-preclean-start] 2015-08-18T19:36:55.165+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-18T19:36:55.165+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-18T19:36:58.104+0000: [CMS-concurrent-abortable-preclean: 0.636/2.939 secs] [Times: user=0.86 sys=0.06, real=2.94 secs] 2015-08-18T19:36:58.145+0000: [CMS-concurrent-sweep-start] 2015-08-18T19:36:58.347+0000: [CMS-concurrent-sweep: 0.201/0.201 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] 2015-08-18T19:36:58.347+0000: [CMS-concurrent-reset-start] 2015-08-18T19:36:58.376+0000: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 2015-08-18 19:37:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1061302, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=899, evicted=23, evictedPerRun=0.025583982467651367 2015-08-18 19:39:53,068 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:39:53,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=61449859, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/76f27328b66a40e8a019fc68e58b118b 2015-08-18 19:39:53,980 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/76f27328b66a40e8a019fc68e58b118b, entries=368274, sequenceid=61449859, filesize=8.8 M 2015-08-18 19:39:53,981 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219640, currentsize=119.85 KB/122728 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 913ms, sequenceid=61449859, compaction requested=true 2015-08-18 19:40:16,987 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.14:10110] 2015-08-18 19:40:17,047 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926531611 with entries=48682, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926816806 2015-08-18 19:40:17,048 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925673668 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925673668 2015-08-18 19:40:17,767 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:40:18,605 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=68193389, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/2f1966121ed5408a81021cff4df6d553 2015-08-18 19:40:18,667 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/2f1966121ed5408a81021cff4df6d553, entries=386738, sequenceid=68193389, filesize=9.0 M 2015-08-18 19:40:18,668 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217968, currentsize=113.70 KB/116424 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 901ms, sequenceid=68193389, compaction requested=true 2015-08-18 19:40:22,165 INFO [sync.4] wal.FSHLog: Slow sync cost: 175 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.9:10110] 2015-08-18 19:40:22,165 INFO [sync.2] wal.FSHLog: Slow sync cost: 219 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.9:10110] 2015-08-18 19:40:22,165 INFO [sync.1] wal.FSHLog: Slow sync cost: 135 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.9:10110] 2015-08-18 19:40:22,165 INFO [sync.3] wal.FSHLog: Slow sync cost: 180 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.9:10110] 2015-08-18 19:40:22,165 INFO [sync.0] wal.FSHLog: Slow sync cost: 155 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.9:10110] 2015-08-18 19:40:47,211 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:40:51,066 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69623053, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/8e859c2d5a2a45fabecea08169e50652 2015-08-18 19:40:51,128 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/8e859c2d5a2a45fabecea08169e50652, entries=365633, sequenceid=69623053, filesize=8.8 M 2015-08-18 19:40:51,130 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134222976, currentsize=638.03 KB/653344 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 3919ms, sequenceid=69623053, compaction requested=true 2015-08-18 19:42:01,135 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:42:02,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67602442, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/cefc43cebf374832a8e20228dda9b6cd 2015-08-18 19:42:02,234 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/e/cefc43cebf374832a8e20228dda9b6cd, entries=348000, sequenceid=67602442, filesize=8.7 M 2015-08-18 19:42:02,236 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134228216, currentsize=156.57 KB/160328 for region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. in 1100ms, sequenceid=67602442, compaction requested=true 2015-08-18 19:42:02,238 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. 2015-08-18 19:42:02,238 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp, totalSize=94.6 M 2015-08-18 19:42:02,245 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:42:09,908 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7. into 9012601a894b4f4f87592d065ba28160(size=94.4 M), total size for store is 4.9 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-18 19:42:09,908 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., storeName=e, fileCount=3, fileSize=94.6 M, priority=20, time=3464050649564034; duration=7sec 2015-08-18 19:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1076427, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=929, evicted=23, evictedPerRun=0.024757804349064827 2015-08-18 19:45:04,468 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926816806 with entries=48726, filesize=121.72 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439927104237 2015-08-18 19:45:04,470 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925962072 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439925962072 2015-08-18 19:45:04,559 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926246435 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439926246435 2015-08-18T19:47:01.168+0000: [CMS-concurrent-mark-start] 2015-08-18T19:47:01.227+0000: [CMS-concurrent-mark: 0.058/0.058 secs] [Times: user=0.29 sys=0.00, real=0.06 secs] 2015-08-18T19:47:01.227+0000: [CMS-concurrent-preclean-start] 2015-08-18T19:47:01.230+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-18T19:47:01.264+0000: [CMS-concurrent-sweep-start] 2015-08-18T19:47:01.484+0000: [CMS-concurrent-sweep: 0.220/0.220 secs] [Times: user=0.25 sys=0.00, real=0.22 secs] 2015-08-18T19:47:01.485+0000: [CMS-concurrent-reset-start] 2015-08-18T19:47:01.514+0000: [CMS-concurrent-reset: 0.030/0.030 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 2015-08-18 19:47:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1076427, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=959, evicted=23, evictedPerRun=0.023983316496014595 2015-08-18 19:47:58,637 INFO [sync.4] wal.FSHLog: Slow sync cost: 128 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.4:10110] 2015-08-18 19:47:58,637 INFO [sync.1] wal.FSHLog: Slow sync cost: 108 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.4:10110] 2015-08-18 19:47:58,637 INFO [sync.0] wal.FSHLog: Slow sync cost: 113 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.4:10110] 2015-08-18 19:47:58,637 INFO [sync.3] wal.FSHLog: Slow sync cost: 133 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.4:10110] 2015-08-18 19:49:51,262 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.4:10110] 2015-08-18 19:49:51,313 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439927104237 with entries=48669, filesize=121.74 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439927391025 2015-08-18 19:49:56,475 INFO [sync.2] wal.FSHLog: Slow sync cost: 249 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.2:10110] 2015-08-18 19:49:56,475 INFO [sync.1] wal.FSHLog: Slow sync cost: 166 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.2:10110] 2015-08-18 19:49:56,475 INFO [sync.0] wal.FSHLog: Slow sync cost: 186 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.2:10110] 2015-08-18 19:49:56,475 INFO [sync.3] wal.FSHLog: Slow sync cost: 245 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.2:10110] 2015-08-18 19:49:56,475 INFO [sync.4] wal.FSHLog: Slow sync cost: 225 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.2:10110] 2015-08-18 19:52:40,420 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.59 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=4, accesses=1076427, hits=26, hitRatio=0.00%, , cachingAccesses=61, cachingHits=26, cachingHitsRatio=42.62%, evictions=989, evicted=23, evictedPerRun=0.023255813866853714 2015-08-18 19:53:22,713 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:53:23,535 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=69689564, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/2c5f12d6eebb46349d39d4d615bafa68 2015-08-18 19:53:23,596 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/2c5f12d6eebb46349d39d4d615bafa68, entries=362878, sequenceid=69689564, filesize=8.8 M 2015-08-18 19:53:23,598 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218848, currentsize=134.28 KB/137504 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 884ms, sequenceid=69689564, compaction requested=true 2015-08-18 19:53:23,600 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-18 19:53:23,600 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=113.6 M 2015-08-18 19:53:23,606 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:53:32,694 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into eb83255aaeb44fccb85f630603f7b458(size=113.4 M), total size for store is 4.6 G. This selection was in queue for 0sec, and took 9sec to execute. 2015-08-18 19:53:32,695 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=113.6 M, priority=21, time=3464732011307703; duration=9sec 2015-08-18 19:54:02,272 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:54:03,106 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=68265214, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/0423ec8bb6a4462b82e7a6bec421fd96 2015-08-18 19:54:03,168 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/0423ec8bb6a4462b82e7a6bec421fd96, entries=386990, sequenceid=68265214, filesize=9.0 M 2015-08-18 19:54:03,169 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222480, currentsize=109.09 KB/111704 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 897ms, sequenceid=68265214, compaction requested=true 2015-08-18 19:54:03,171 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-18 19:54:03,172 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=80.3 M 2015-08-18 19:54:03,177 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=4, currentSize=4812208, freeSize=4284807504, maxSize=4289619712, heapSize=4812208, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-18 19:54:09,873 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 34138163fa044ae0b795a016731fa688(size=80.1 M), total size for store is 4.8 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-18 19:54:09,874 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=80.3 M, priority=20, time=3464771582954856; duration=6sec 2015-08-18 19:54:35,901 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.2:10110] 2015-08-18 19:54:35,960 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439927391025 with entries=48841, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1439927675714 2015-08-18 19:54:44,608 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x09\x00\x00\x00\x00\x00\x00\x00\x02/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet1/intfCounter/lastClear,1438987184868.7e981f7e8b0732d7dfcd2e63005063c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-18 19:54:45,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=67668054, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/7e981f7e8b0732d7dfcd2e63005063c7/.tmp/04701e83844f46d0b3b5884e797d4270 2015-08-22 23:48:05,101 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287005428 with entries=49973, filesize=121.65 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287284909 2015-08-22 23:50:51,928 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-22 23:50:52,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99510609, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/9e78b354ce2744d1a02f9bd563b96382 2015-08-22 23:50:52,779 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/9e78b354ce2744d1a02f9bd563b96382, entries=384663, sequenceid=99510609, filesize=8.9 M 2015-08-22 23:50:52,780 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217944, currentsize=96.07 KB/98376 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 852ms, sequenceid=99510609, compaction requested=true 2015-08-22 23:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=19966428, hits=334, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=36989, evicted=159, evictedPerRun=0.004298575222492218 2015-08-22 23:52:46,056 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287284909 with entries=50881, filesize=121.75 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287565670 2015-08-22 23:53:44,833 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-22 23:53:45,695 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101329575, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/6ceab097ed4f4829aae82a23f403a5b2 2015-08-22 23:53:45,730 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/6ceab097ed4f4829aae82a23f403a5b2, entries=365622, sequenceid=101329575, filesize=8.7 M 2015-08-22 23:53:45,732 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219184, currentsize=225.45 KB/230864 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 899ms, sequenceid=101329575, compaction requested=true 2015-08-22 23:53:45,734 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-22 23:53:45,734 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=43.2 M 2015-08-22 23:53:45,741 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-22 23:53:49,545 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 0f5f230d3b4743f8b6a0997ab9289b13(size=43.0 M), total size for store is 8.5 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-22 23:53:49,545 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=43.2 M, priority=19, time=3824754145581300; duration=3sec 2015-08-22 23:54:36,216 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed. 2015-08-22 23:54:36,992 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97586964, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/76e90b2d55024f41af44b9c767cf5525 2015-08-22 23:54:37,028 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/76e90b2d55024f41af44b9c767cf5525, entries=322172, sequenceid=97586964, filesize=8.6 M 2015-08-22 23:54:37,029 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134238992, currentsize=63.52 KB/65048 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 813ms, sequenceid=97586964, compaction requested=true 2015-08-22 23:54:37,031 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-22 23:54:37,031 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=76.5 M 2015-08-22 23:54:37,040 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-22 23:54:43,468 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 325206a43ce047d8aadf03bdc7dd0cff(size=76.2 M), total size for store is 7.2 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-22 23:54:43,468 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=76.5 M, priority=20, time=3824805442658223; duration=6sec 2015-08-22 23:56:19,777 INFO [sync.4] wal.FSHLog: Slow sync cost: 147 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-22 23:56:19,777 INFO [sync.3] wal.FSHLog: Slow sync cost: 151 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-22 23:56:19,778 INFO [sync.1] wal.FSHLog: Slow sync cost: 213 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-22 23:56:19,778 INFO [sync.2] wal.FSHLog: Slow sync cost: 173 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-22 23:56:19,778 INFO [sync.0] wal.FSHLog: Slow sync cost: 133 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-22 23:57:23,951 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 116 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-22 23:57:23,985 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287565670 with entries=50694, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287843733 2015-08-22 23:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=19985964, hits=334, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37019, evicted=159, evictedPerRun=0.004295091610401869 2015-08-22 23:58:29,647 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-22 23:58:30,883 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90313337, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/613379429ecd44eeba277679ef139da3 2015-08-22 23:58:30,936 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/613379429ecd44eeba277679ef139da3, entries=367269, sequenceid=90313337, filesize=8.7 M 2015-08-22 23:58:30,937 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223984, currentsize=124.95 KB/127944 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 1290ms, sequenceid=90313337, compaction requested=true 2015-08-22 23:58:30,939 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-22 23:58:30,939 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=60.6 M 2015-08-22 23:58:30,947 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-22 23:58:36,417 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into b4cca8f41a19438484232a1a5f81a849(size=60.4 M), total size for store is 6.1 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-22 23:58:36,417 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=60.6 M, priority=19, time=3825039350497076; duration=5sec 2015-08-22T23:59:17.251+0000: [CMS-concurrent-mark-start] 2015-08-22T23:59:17.308+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.30 sys=0.00, real=0.06 secs] 2015-08-22T23:59:17.308+0000: [CMS-concurrent-preclean-start] 2015-08-22T23:59:17.311+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-22T23:59:17.347+0000: [CMS-concurrent-sweep-start] 2015-08-22T23:59:17.572+0000: [CMS-concurrent-sweep: 0.225/0.225 secs] [Times: user=0.26 sys=0.00, real=0.23 secs] 2015-08-22T23:59:17.572+0000: [CMS-concurrent-reset-start] 2015-08-22T23:59:17.592+0000: [CMS-concurrent-reset: 0.020/0.020 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 00:02:05,330 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.3:10110] 2015-08-23 00:02:05,389 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287843733 with entries=50022, filesize=121.75 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288125151 2015-08-23 00:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=19995344, hits=334, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37049, evicted=159, evictedPerRun=0.004291613586246967 2015-08-23 00:04:49,193 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:04:50,023 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99583976, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/3b21109e72584b76befb90495b463fba 2015-08-23 00:04:50,084 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/3b21109e72584b76befb90495b463fba, entries=384419, sequenceid=99583976, filesize=9.0 M 2015-08-23 00:04:50,085 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219816, currentsize=152.39 KB/156048 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 892ms, sequenceid=99583976, compaction requested=true 2015-08-23 00:04:50,088 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 00:04:50,088 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=61.9 M 2015-08-23 00:04:50,094 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:04:55,674 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 68fd4b391a9c4ccf868acba52655595e(size=61.7 M), total size for store is 7.7 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 00:04:55,674 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=61.9 M, priority=20, time=3825418499276007; duration=5sec 2015-08-23 00:06:39,410 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:06:40,235 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101398241, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/4917c1c9509b49b28be09f0d451171d7 2015-08-23 00:06:40,272 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/4917c1c9509b49b28be09f0d451171d7, entries=365296, sequenceid=101398241, filesize=8.7 M 2015-08-23 00:06:40,273 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219888, currentsize=137.09 KB/140376 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 863ms, sequenceid=101398241, compaction requested=true 2015-08-23 00:06:44,503 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288125151 with entries=49400, filesize=121.61 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288404310 2015-08-23 00:07:06,056 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:07:06,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97643962, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/bf2a8a715f3a47398b4dc7604740e912 2015-08-23 00:07:06,874 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/bf2a8a715f3a47398b4dc7604740e912, entries=323028, sequenceid=97643962, filesize=8.6 M 2015-08-23 00:07:06,875 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221464, currentsize=82.49 KB/84472 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 819ms, sequenceid=97643962, compaction requested=true 2015-08-23 00:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=20004494, hits=334, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37079, evicted=159, evictedPerRun=0.004288141615688801 2015-08-23 00:11:22,183 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288404310 with entries=48542, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288682033 2015-08-23 00:11:27,358 INFO [sync.4] wal.FSHLog: Slow sync cost: 129 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.7:10110] 2015-08-23 00:11:27,358 INFO [sync.3] wal.FSHLog: Slow sync cost: 149 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.7:10110] 2015-08-23 00:11:27,359 INFO [sync.2] wal.FSHLog: Slow sync cost: 194 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.7:10110] 2015-08-23 00:11:27,359 INFO [sync.1] wal.FSHLog: Slow sync cost: 206 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.7:10110] 2015-08-23 00:11:27,359 INFO [sync.0] wal.FSHLog: Slow sync cost: 111 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.7:10110] 2015-08-23 00:11:30,309 INFO [sync.2] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.7:10110] 2015-08-23T00:12:37.630+0000: [CMS-concurrent-mark-start] 2015-08-23T00:12:37.718+0000: [CMS-concurrent-mark: 0.088/0.088 secs] [Times: user=0.46 sys=0.00, real=0.09 secs] 2015-08-23T00:12:37.718+0000: [CMS-concurrent-preclean-start] 2015-08-23T00:12:37.721+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T00:12:37.721+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23 00:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=20004494, hits=334, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37109, evicted=159, evictedPerRun=0.004284674767404795 2015-08-23T00:12:40.841+0000: [CMS-concurrent-abortable-preclean: 0.615/3.120 secs] [Times: user=0.86 sys=0.05, real=3.12 secs] 2015-08-23T00:12:40.878+0000: [CMS-concurrent-sweep-start] 2015-08-23T00:12:41.082+0000: [CMS-concurrent-sweep: 0.205/0.205 secs] [Times: user=0.23 sys=0.00, real=0.21 secs] 2015-08-23T00:12:41.083+0000: [CMS-concurrent-reset-start] 2015-08-23T00:12:41.103+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 00:13:56,866 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:13:57,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90386407, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/8d32a90cea8c4ba389eb6e1263b7ba7a 2015-08-23 00:13:57,701 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/8d32a90cea8c4ba389eb6e1263b7ba7a, entries=366749, sequenceid=90386407, filesize=8.7 M 2015-08-23 00:13:57,702 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219440, currentsize=123.30 KB/126264 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 837ms, sequenceid=90386407, compaction requested=true 2015-08-23 00:13:57,706 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 00:13:57,706 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.HStore: Starting compaction of 5 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=6.1 G 2015-08-23 00:13:57,718 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:16:02,380 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288682033 with entries=49001, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288962210 2015-08-23 00:17:40,424 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=20397709, hits=334, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37139, evicted=159, evictedPerRun=0.004281213972717524 2015-08-23 00:18:43,144 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:18:44,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99655752, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/c557b91915eb474094767ee42d913efe 2015-08-23 00:18:44,075 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/c557b91915eb474094767ee42d913efe, entries=384789, sequenceid=99655752, filesize=8.9 M 2015-08-23 00:18:44,076 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223496, currentsize=163.25 KB/167168 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 933ms, sequenceid=99655752, compaction requested=true 2015-08-23 00:19:32,805 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:19:33,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97699585, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/4a6ec1e85bd546a7afdce3022190a4af 2015-08-23 00:19:33,647 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/4a6ec1e85bd546a7afdce3022190a4af, entries=321868, sequenceid=97699585, filesize=8.6 M 2015-08-23 00:19:33,647 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218864, currentsize=171.84 KB/175968 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 842ms, sequenceid=97699585, compaction requested=true 2015-08-23 00:19:33,649 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 00:19:33,649 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=93.4 M 2015-08-23 00:19:33,657 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:19:36,156 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:19:36,245 INFO [MemStoreFlusher.1] compress.CodecPool: Got brand-new compressor [.snappy] 2015-08-23 00:19:36,246 INFO [MemStoreFlusher.1] compress.CodecPool: Got brand-new compressor [.snappy] 2015-08-23 00:19:36,995 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101466124, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/fa2825b43cc6463e92c9fb04df85941f 2015-08-23 00:19:37,047 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/fa2825b43cc6463e92c9fb04df85941f, entries=365254, sequenceid=101466124, filesize=8.7 M 2015-08-23 00:19:37,048 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223872, currentsize=109.94 KB/112576 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 892ms, sequenceid=101466124, compaction requested=true 2015-08-23 00:19:41,745 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 77d49c2f303c4f4a822b33bc3361d837(size=93.2 M), total size for store is 7.2 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-23 00:19:41,747 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=93.4 M, priority=20, time=3826302060732094; duration=8sec 2015-08-23 00:19:41,749 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 00:19:41,749 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=60.5 M 2015-08-23 00:19:41,758 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:19:47,263 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 581cc4619f2c431e9d957b3942681675(size=60.2 M), total size for store is 8.5 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 00:19:47,263 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=60.5 M, priority=19, time=3826310160896430; duration=5sec 2015-08-23 00:20:42,361 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288962210 with entries=49546, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289242213 2015-08-23 00:20:47,542 INFO [sync.0] wal.FSHLog: Slow sync cost: 195 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.6:10110] 2015-08-23 00:20:47,542 INFO [sync.1] wal.FSHLog: Slow sync cost: 193 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.6:10110] 2015-08-23 00:20:47,543 INFO [sync.2] wal.FSHLog: Slow sync cost: 173 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.6:10110] 2015-08-23 00:20:47,543 INFO [sync.4] wal.FSHLog: Slow sync cost: 138 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.6:10110] 2015-08-23 00:20:47,543 INFO [sync.3] wal.FSHLog: Slow sync cost: 158 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.6:10110] 2015-08-23T00:21:29.684+0000: [CMS-concurrent-mark-start] 2015-08-23T00:21:29.737+0000: [CMS-concurrent-mark: 0.052/0.052 secs] [Times: user=0.32 sys=0.00, real=0.05 secs] 2015-08-23T00:21:29.737+0000: [CMS-concurrent-preclean-start] 2015-08-23T00:21:29.741+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-23T00:21:29.741+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T00:21:30.010+0000: [CMS-concurrent-abortable-preclean: 0.036/0.270 secs] [Times: user=0.55 sys=0.00, real=0.27 secs] 2015-08-23T00:21:30.041+0000: [CMS-concurrent-sweep-start] 2015-08-23T00:21:30.283+0000: [CMS-concurrent-sweep: 0.234/0.242 secs] [Times: user=0.57 sys=0.00, real=0.25 secs] 2015-08-23T00:21:30.283+0000: [CMS-concurrent-reset-start] 2015-08-23T00:21:30.304+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 2015-08-23 00:22:30,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 21095 2015-08-23 00:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.90 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=27, accesses=20988133, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37169, evicted=159, evictedPerRun=0.004277758300304413 2015-08-23 00:22:40,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 10034 2015-08-23 00:22:50,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 15051 2015-08-23 00:22:51,971 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.64 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:22:52,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57252558, memsize=56.6 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/51c01a17b34e40878bce7ecc407e404a 2015-08-23 00:22:52,418 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/51c01a17b34e40878bce7ecc407e404a, entries=196974, sequenceid=57252558, filesize=3.6 M 2015-08-23 00:22:52,419 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~56.64 MB/59388640, currentsize=8.22 KB/8416 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 448ms, sequenceid=57252558, compaction requested=true 2015-08-23 00:22:52,420 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. 2015-08-23 00:22:52,420 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp, totalSize=72.7 M 2015-08-23 00:22:52,426 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=27, currentSize=6182288, freeSize=4283437424, maxSize=4289619712, heapSize=6182288, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:22:59,384 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into 9688e839a3ff41008e772e5062861f00(size=72.7 M), total size for store is 1.8 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 00:22:59,384 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., storeName=e, fileCount=3, fileSize=72.7 M, priority=20, time=3826500831798174; duration=6sec 2015-08-23 00:23:05,042 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.HStore: Completed compaction of 5 (all) file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 6973529dd7514ca3aaa7bd6f1fdd160f(size=6.1 G), total size for store is 6.1 G. This selection was in queue for 0sec, and took 9mins, 7sec to execute. 2015-08-23 00:23:05,044 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-longCompactions-1439917700325] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=5, fileSize=6.1 G, priority=20, time=3825966117113896; duration=9mins, 7sec 2015-08-23 00:25:21,225 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289242213 with entries=50113, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289521117 2015-08-23 00:25:21,227 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440285605235 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440285605235 2015-08-23 00:25:21,258 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440285888152 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440285888152 2015-08-23 00:25:21,291 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440286166528 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440286166528 2015-08-23 00:25:21,325 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440286445071 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440286445071 2015-08-23 00:25:21,358 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440286725912 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440286725912 2015-08-23 00:25:21,391 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287005428 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287005428 2015-08-23 00:25:21,425 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287284909 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287284909 2015-08-23 00:25:21,458 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287565670 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287565670 2015-08-23 00:25:21,491 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287843733 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440287843733 2015-08-23 00:25:21,525 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288125151 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288125151 2015-08-23 00:25:21,558 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288404310 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288404310 2015-08-23 00:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.24 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=15, accesses=21047286, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37199, evicted=171, evictedPerRun=0.00459689786657691 2015-08-23 00:29:27,606 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:29:28,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90460935, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/413e3a875e4a41d680090a0f4ff6f268 2015-08-23 00:29:28,403 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/413e3a875e4a41d680090a0f4ff6f268, entries=367224, sequenceid=90460935, filesize=8.7 M 2015-08-23 00:29:28,405 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220984, currentsize=166.77 KB/170768 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 798ms, sequenceid=90460935, compaction requested=false 2015-08-23 00:29:59,629 INFO [sync.1] wal.FSHLog: Slow sync cost: 105 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.15:10110] 2015-08-23 00:30:02,122 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289521117 with entries=49699, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289802052 2015-08-23 00:30:02,124 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288682033 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288682033 2015-08-23 00:32:06,128 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:32:06,836 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97756289, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/65d5aef1acfe4ffc8611077a42460f2d 2015-08-23 00:32:06,863 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/65d5aef1acfe4ffc8611077a42460f2d, entries=323612, sequenceid=97756289, filesize=8.6 M 2015-08-23 00:32:06,864 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219080, currentsize=150.88 KB/154496 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 736ms, sequenceid=97756289, compaction requested=true 2015-08-23 00:32:31,390 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:32:32,163 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101533438, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/6cdc03f81a4d4f6db492eba3c1f5bfda 2015-08-23 00:32:32,182 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/6cdc03f81a4d4f6db492eba3c1f5bfda, entries=365028, sequenceid=101533438, filesize=8.7 M 2015-08-23 00:32:32,183 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218352, currentsize=128.12 KB/131192 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 793ms, sequenceid=101533438, compaction requested=true 2015-08-23 00:32:32,185 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 00:32:32,186 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 4 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=387.4 M 2015-08-23 00:32:32,191 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=15, currentSize=5495552, freeSize=4284124160, maxSize=4289619712, heapSize=5495552, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:32:38,913 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:32:39,817 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99727589, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/ca7c9fa39f8d4768a209d41b3fd9bba2 2015-08-23 00:32:39,833 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/ca7c9fa39f8d4768a209d41b3fd9bba2, entries=385133, sequenceid=99727589, filesize=8.9 M 2015-08-23 00:32:39,834 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219584, currentsize=184.98 KB/189424 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 921ms, sequenceid=99727589, compaction requested=true 2015-08-23 00:32:40,432 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.24 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=15, accesses=21065370, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37229, evicted=171, evictedPerRun=0.004593193531036377 2015-08-23 00:33:04,000 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 4 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into aa627ad3e8234a37ad94c0c2d9d065fd(size=387.9 M), total size for store is 8.5 G. This selection was in queue for 0sec, and took 31sec to execute. 2015-08-23 00:33:04,000 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=4, fileSize=387.4 M, priority=20, time=3827080597011622; duration=31sec 2015-08-23 00:33:04,002 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 00:33:04,002 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=79.5 M 2015-08-23 00:33:04,009 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:33:10,476 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 1b760d0978fc47138203455602510950(size=79.3 M), total size for store is 7.8 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 00:33:10,476 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=79.5 M, priority=20, time=3827112413561331; duration=6sec 2015-08-23 00:34:41,395 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289802052 with entries=47433, filesize=121.61 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290081309 2015-08-23 00:34:41,398 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288962210 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440288962210 2015-08-23T00:35:25.350+0000: [CMS-concurrent-mark-start] 2015-08-23T00:35:25.395+0000: [CMS-concurrent-mark: 0.045/0.045 secs] [Times: user=0.23 sys=0.00, real=0.05 secs] 2015-08-23T00:35:25.395+0000: [CMS-concurrent-preclean-start] 2015-08-23T00:35:25.398+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-23T00:35:25.432+0000: [CMS-concurrent-sweep-start] 2015-08-23T00:35:25.668+0000: [CMS-concurrent-sweep: 0.236/0.236 secs] [Times: user=0.26 sys=0.00, real=0.23 secs] 2015-08-23T00:35:25.668+0000: [CMS-concurrent-reset-start] 2015-08-23T00:35:25.690+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 00:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21119361, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37259, evicted=173, evictedPerRun=0.004643173422664404 2015-08-23 00:39:19,342 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290081309 with entries=49025, filesize=121.65 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290359249 2015-08-23 00:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21119361, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37289, evicted=173, evictedPerRun=0.004639437887817621 2015-08-23 00:42:43,777 INFO [sync.4] wal.FSHLog: Slow sync cost: 168 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 00:42:43,777 INFO [sync.3] wal.FSHLog: Slow sync cost: 188 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 00:42:43,777 INFO [sync.2] wal.FSHLog: Slow sync cost: 213 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 00:42:43,777 INFO [sync.1] wal.FSHLog: Slow sync cost: 229 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 00:42:43,778 INFO [sync.0] wal.FSHLog: Slow sync cost: 154 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 00:44:01,822 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290359249 with entries=49487, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290641645 2015-08-23 00:44:32,517 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.08 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:44:33,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97811910, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/7e2bdfa7895a434082d37b0201a0b491 2015-08-23 00:44:33,318 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/7e2bdfa7895a434082d37b0201a0b491, entries=321804, sequenceid=97811910, filesize=8.6 M 2015-08-23 00:44:33,320 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134306632, currentsize=100.50 KB/102912 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 803ms, sequenceid=97811910, compaction requested=true 2015-08-23 00:44:33,321 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 00:44:33,321 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=110.3 M 2015-08-23 00:44:33,329 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 00:44:42,484 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 8eb8ce505bb94c2f817e7fa48b05c0f1(size=110.1 M), total size for store is 7.2 G. This selection was in queue for 0sec, and took 9sec to execute. 2015-08-23 00:44:42,484 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=110.3 M, priority=20, time=3827801732912576; duration=9sec 2015-08-23 00:44:59,445 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:45:00,244 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90534112, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/33b36171d11643fe9ee2322a1274375f 2015-08-23 00:45:00,281 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/33b36171d11643fe9ee2322a1274375f, entries=367120, sequenceid=90534112, filesize=8.7 M 2015-08-23 00:45:00,282 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221736, currentsize=104.07 KB/106568 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 837ms, sequenceid=90534112, compaction requested=true 2015-08-23 00:45:26,156 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:45:26,947 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101601258, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/cd7fcf79930c4b11a731928449d0ad6f 2015-08-23 00:45:26,983 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/cd7fcf79930c4b11a731928449d0ad6f, entries=364848, sequenceid=101601258, filesize=8.7 M 2015-08-23 00:45:26,985 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222768, currentsize=95.17 KB/97456 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 829ms, sequenceid=101601258, compaction requested=true 2015-08-23 00:46:32,530 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:46:33,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99799471, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/eaccf87dfc1e4d18b155fa953e969cd4 2015-08-23 00:46:33,382 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/eaccf87dfc1e4d18b155fa953e969cd4, entries=385149, sequenceid=99799471, filesize=8.9 M 2015-08-23 00:46:33,384 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219856, currentsize=125.62 KB/128632 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 854ms, sequenceid=99799471, compaction requested=true 2015-08-23 00:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21137857, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37319, evicted=173, evictedPerRun=0.0046357084065675735 2015-08-23 00:48:38,909 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.7:10110, 172.24.32.11:10110] 2015-08-23 00:48:38,959 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290641645 with entries=48632, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290918667 2015-08-23T00:50:27.009+0000: [CMS-concurrent-mark-start] 2015-08-23T00:50:27.064+0000: [CMS-concurrent-mark: 0.055/0.055 secs] [Times: user=0.28 sys=0.00, real=0.06 secs] 2015-08-23T00:50:27.064+0000: [CMS-concurrent-preclean-start] 2015-08-23T00:50:27.068+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T00:50:27.068+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T00:50:29.998+0000: [CMS-concurrent-abortable-preclean: 0.726/2.929 secs] [Times: user=0.91 sys=0.04, real=2.93 secs] 2015-08-23T00:50:30.029+0000: [CMS-concurrent-sweep-start] 2015-08-23T00:50:30.258+0000: [CMS-concurrent-sweep: 0.228/0.228 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T00:50:30.258+0000: [CMS-concurrent-reset-start] 2015-08-23T00:50:30.279+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 00:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21137857, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37349, evicted=173, evictedPerRun=0.004631984978914261 2015-08-23 00:53:18,271 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290918667 with entries=49542, filesize=121.68 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291198038 2015-08-23 00:57:04,116 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.05 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:57:04,937 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97868070, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/9422eb6c6c9245a1af0e7fd06756f7f1 2015-08-23 00:57:04,998 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/9422eb6c6c9245a1af0e7fd06756f7f1, entries=323288, sequenceid=97868070, filesize=8.6 M 2015-08-23 00:57:04,999 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134270872, currentsize=76.03 KB/77856 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 883ms, sequenceid=97868070, compaction requested=true 2015-08-23 00:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21137857, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37379, evicted=173, evictedPerRun=0.004628267139196396 2015-08-23 00:57:59,809 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.2:10110] 2015-08-23 00:57:59,869 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291198038 with entries=49618, filesize=121.70 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291479627 2015-08-23 00:58:04,997 INFO [sync.1] wal.FSHLog: Slow sync cost: 153 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.14:10110] 2015-08-23 00:58:04,997 INFO [sync.0] wal.FSHLog: Slow sync cost: 187 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.14:10110] 2015-08-23 00:58:04,997 INFO [sync.4] wal.FSHLog: Slow sync cost: 208 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.14:10110] 2015-08-23 00:58:04,997 INFO [sync.3] wal.FSHLog: Slow sync cost: 229 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.14:10110] 2015-08-23 00:58:04,997 INFO [sync.2] wal.FSHLog: Slow sync cost: 148 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.14:10110] 2015-08-23 00:58:21,135 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 00:58:22,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101668945, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/34262c4418d24aa0abafba68a593b0bc 2015-08-23 00:58:22,102 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/34262c4418d24aa0abafba68a593b0bc, entries=365224, sequenceid=101668945, filesize=8.7 M 2015-08-23 00:58:22,103 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222344, currentsize=167.14 KB/171152 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 968ms, sequenceid=101668945, compaction requested=true 2015-08-23 01:00:29,388 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:00:30,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99871592, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/1d55ef4fdd654bf1ae85ce04ac95c756 2015-08-23 01:00:30,220 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/1d55ef4fdd654bf1ae85ce04ac95c756, entries=384567, sequenceid=99871592, filesize=8.9 M 2015-08-23 01:00:30,221 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222184, currentsize=122.21 KB/125144 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 832ms, sequenceid=99871592, compaction requested=true 2015-08-23 01:00:30,222 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 01:00:30,222 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=97.1 M 2015-08-23 01:00:30,229 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:00:32,386 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:00:33,268 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90608585, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/d7571815054740c2922f6ee213c2f31c 2015-08-23 01:00:33,304 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/d7571815054740c2922f6ee213c2f31c, entries=367799, sequenceid=90608585, filesize=8.7 M 2015-08-23 01:00:33,306 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223504, currentsize=93.08 KB/95312 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 920ms, sequenceid=90608585, compaction requested=true 2015-08-23 01:00:38,577 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into f3c582d1de8d455a9a453fd3c4ac0396(size=96.9 M), total size for store is 7.8 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-23 01:00:38,578 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=97.1 M, priority=20, time=3828758633739320; duration=8sec 2015-08-23 01:00:38,580 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 01:00:38,580 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=26.2 M 2015-08-23 01:00:38,591 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:00:41,185 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 517e36d525c34d03b2794b7262bcc02e(size=25.9 M), total size for store is 6.1 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 01:00:41,185 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=26.2 M, priority=21, time=3828766991465720; duration=2sec 2015-08-23 01:02:38,874 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291479627 with entries=49514, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291758711 2015-08-23 01:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21156265, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37409, evicted=173, evictedPerRun=0.004624555818736553 2015-08-23T01:03:47.383+0000: [CMS-concurrent-mark-start] 2015-08-23T01:03:47.439+0000: [CMS-concurrent-mark: 0.056/0.056 secs] [Times: user=0.28 sys=0.00, real=0.05 secs] 2015-08-23T01:03:47.439+0000: [CMS-concurrent-preclean-start] 2015-08-23T01:03:47.442+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T01:03:47.474+0000: [CMS-concurrent-sweep-start] 2015-08-23T01:03:47.697+0000: [CMS-concurrent-sweep: 0.223/0.223 secs] [Times: user=0.25 sys=0.00, real=0.22 secs] 2015-08-23T01:03:47.697+0000: [CMS-concurrent-reset-start] 2015-08-23T01:03:47.718+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 01:05:46,509 INFO [sync.4] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.11:10110] 2015-08-23 01:05:46,509 INFO [sync.0] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.11:10110] 2015-08-23 01:07:17,288 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291758711 with entries=49584, filesize=121.71 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292037132 2015-08-23 01:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21156265, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37439, evicted=173, evictedPerRun=0.004620850086212158 2015-08-23 01:09:31,069 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:09:31,861 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97923885, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/5f1d58e5a29c44fba128e158bdcb9a06 2015-08-23 01:09:31,897 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/5f1d58e5a29c44fba128e158bdcb9a06, entries=321353, sequenceid=97923885, filesize=8.6 M 2015-08-23 01:09:31,898 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134224360, currentsize=128.81 KB/131904 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 829ms, sequenceid=97923885, compaction requested=true 2015-08-23 01:09:31,901 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 01:09:31,901 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=127.2 M 2015-08-23 01:09:31,908 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:09:42,520 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 911796d912284e60b0fb9d8fc38a7497(size=127.0 M), total size for store is 7.2 G. This selection was in queue for 0sec, and took 10sec to execute. 2015-08-23 01:09:42,521 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=127.2 M, priority=20, time=3829300312604834; duration=10sec 2015-08-23 01:11:20,253 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:11:21,056 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101737896, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/e63baedd0e134866aad9dac939c847da 2015-08-23 01:11:21,092 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/e63baedd0e134866aad9dac939c847da, entries=366644, sequenceid=101737896, filesize=8.7 M 2015-08-23 01:11:21,094 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219896, currentsize=77.97 KB/79840 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 841ms, sequenceid=101737896, compaction requested=true 2015-08-23 01:11:21,096 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 01:11:21,096 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=26.1 M 2015-08-23 01:11:21,102 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:11:23,623 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 7bd9e110d2dc41f6bd37a6876bda3d89(size=25.8 M), total size for store is 8.6 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 01:11:23,623 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=26.1 M, priority=20, time=3829409507410624; duration=2sec 2015-08-23 01:11:58,169 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292037132 with entries=50249, filesize=121.70 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292318012 2015-08-23 01:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21181659, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37469, evicted=173, evictedPerRun=0.004617150407284498 2015-08-23 01:14:27,312 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:14:28,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99944670, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/39a92ce578204b10817b1c742dc82919 2015-08-23 01:14:28,681 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/39a92ce578204b10817b1c742dc82919, entries=384527, sequenceid=99944670, filesize=8.9 M 2015-08-23 01:14:28,682 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219504, currentsize=202.55 KB/207408 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 1370ms, sequenceid=99944670, compaction requested=true 2015-08-23 01:16:02,146 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:16:02,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90683311, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/223559a95a344a25b6468b45800c36d2 2015-08-23 01:16:03,029 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/223559a95a344a25b6468b45800c36d2, entries=367058, sequenceid=90683311, filesize=8.8 M 2015-08-23 01:16:03,030 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223720, currentsize=85.89 KB/87952 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 884ms, sequenceid=90683311, compaction requested=true 2015-08-23 01:16:36,214 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.3:10110, 172.24.32.6:10110] 2015-08-23 01:16:36,273 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292318012 with entries=50193, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292596032 2015-08-23T01:17:13.284+0000: [CMS-concurrent-mark-start] 2015-08-23T01:17:13.340+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.30 sys=0.00, real=0.05 secs] 2015-08-23T01:17:13.341+0000: [CMS-concurrent-preclean-start] 2015-08-23T01:17:13.344+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T01:17:13.344+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T01:17:15.961+0000: [CMS-concurrent-abortable-preclean: 0.614/2.617 secs] [Times: user=0.82 sys=0.03, real=2.62 secs] 2015-08-23T01:17:15.999+0000: [CMS-concurrent-sweep-start] 2015-08-23T01:17:16.224+0000: [CMS-concurrent-sweep: 0.225/0.225 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T01:17:16.224+0000: [CMS-concurrent-reset-start] 2015-08-23T01:17:16.245+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 01:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21181659, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37499, evicted=173, evictedPerRun=0.004613456316292286 2015-08-23 01:21:15,094 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292596032 with entries=49608, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292874930 2015-08-23 01:21:20,260 INFO [sync.3] wal.FSHLog: Slow sync cost: 190 ms, current pipeline: [172.24.32.16:10110, 172.24.32.4:10110, 172.24.32.13:10110] 2015-08-23 01:21:20,260 INFO [sync.2] wal.FSHLog: Slow sync cost: 207 ms, current pipeline: [172.24.32.16:10110, 172.24.32.4:10110, 172.24.32.13:10110] 2015-08-23 01:21:20,260 INFO [sync.4] wal.FSHLog: Slow sync cost: 155 ms, current pipeline: [172.24.32.16:10110, 172.24.32.4:10110, 172.24.32.13:10110] 2015-08-23 01:21:20,260 INFO [sync.0] wal.FSHLog: Slow sync cost: 135 ms, current pipeline: [172.24.32.16:10110, 172.24.32.4:10110, 172.24.32.13:10110] 2015-08-23 01:22:02,317 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:22:03,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=97981253, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/1c44b006ded6445daba4af72cd27f6b0 2015-08-23 01:22:03,110 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/1c44b006ded6445daba4af72cd27f6b0, entries=323182, sequenceid=97981253, filesize=8.6 M 2015-08-23 01:22:03,111 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134224256, currentsize=76.04 KB/77864 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 794ms, sequenceid=97981253, compaction requested=true 2015-08-23 01:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21181659, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37529, evicted=173, evictedPerRun=0.004609768278896809 2015-08-23 01:23:00,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 14728 2015-08-23 01:23:10,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 8810 2015-08-23 01:23:15,603 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.76 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:23:16,040 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57341268, memsize=56.8 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/d555cad4fe4c424286a86ed4f50fb88d 2015-08-23 01:23:16,077 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/d555cad4fe4c424286a86ed4f50fb88d, entries=196965, sequenceid=57341268, filesize=3.6 M 2015-08-23 01:23:16,078 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~56.76 MB/59512344, currentsize=5.78 KB/5920 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 475ms, sequenceid=57341268, compaction requested=true 2015-08-23 01:24:15,234 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:24:16,055 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101806988, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/978f7cdf6a064f1a94f06fdecdf95d4e 2015-08-23 01:24:16,091 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/978f7cdf6a064f1a94f06fdecdf95d4e, entries=365275, sequenceid=101806988, filesize=8.7 M 2015-08-23 01:24:16,092 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134230576, currentsize=426.33 KB/436560 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 858ms, sequenceid=101806988, compaction requested=true 2015-08-23 01:25:56,417 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292874930 with entries=49765, filesize=121.70 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293156250 2015-08-23 01:25:56,419 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289242213 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289242213 2015-08-23 01:25:56,483 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289521117 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289521117 2015-08-23 01:25:56,550 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289802052 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440289802052 2015-08-23 01:25:56,617 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290081309 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290081309 2015-08-23 01:25:56,683 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290359249 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290359249 2015-08-23 01:25:56,750 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290641645 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290641645 2015-08-23 01:25:56,817 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290918667 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440290918667 2015-08-23 01:25:56,883 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291198038 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291198038 2015-08-23 01:25:56,950 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291479627 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291479627 2015-08-23 01:25:57,017 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291758711 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440291758711 2015-08-23 01:25:57,083 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292037132 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292037132 2015-08-23 01:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21181659, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37559, evicted=173, evictedPerRun=0.004606086295098066 2015-08-23 01:28:24,010 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:28:24,840 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100017553, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/8b9d532ed0264d02925e9cf162c05088 2015-08-23 01:28:24,876 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/8b9d532ed0264d02925e9cf162c05088, entries=384836, sequenceid=100017553, filesize=8.9 M 2015-08-23 01:28:24,877 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217760, currentsize=100.91 KB/103336 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 867ms, sequenceid=100017553, compaction requested=true 2015-08-23 01:28:24,879 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 01:28:24,879 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=114.7 M 2015-08-23 01:28:24,886 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:28:34,383 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 7ae471c8cc8d474cbc0b62774f8b101e(size=114.6 M), total size for store is 7.8 G. This selection was in queue for 0sec, and took 9sec to execute. 2015-08-23 01:28:34,383 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=114.7 M, priority=20, time=3830433290841664; duration=9sec 2015-08-23T01:30:30.443+0000: [CMS-concurrent-mark-start] 2015-08-23T01:30:30.512+0000: [CMS-concurrent-mark: 0.069/0.069 secs] [Times: user=0.35 sys=0.00, real=0.07 secs] 2015-08-23T01:30:30.512+0000: [CMS-concurrent-preclean-start] 2015-08-23T01:30:30.515+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T01:30:30.549+0000: [CMS-concurrent-sweep-start] 2015-08-23T01:30:30.766+0000: [CMS-concurrent-sweep: 0.217/0.217 secs] [Times: user=0.23 sys=0.00, real=0.22 secs] 2015-08-23T01:30:30.766+0000: [CMS-concurrent-reset-start] 2015-08-23T01:30:30.788+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 01:30:35,888 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293156250 with entries=48576, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293435728 2015-08-23 01:31:29,431 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:31:30,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90756910, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/472f2bf5f8d4496bb5966f53629b199a 2015-08-23 01:31:30,362 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/472f2bf5f8d4496bb5966f53629b199a, entries=366696, sequenceid=90756910, filesize=8.7 M 2015-08-23 01:31:30,363 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134226968, currentsize=110.74 KB/113400 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 932ms, sequenceid=90756910, compaction requested=true 2015-08-23 01:31:30,365 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 01:31:30,365 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=43.4 M 2015-08-23 01:31:30,373 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:31:34,310 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 8c9524f9f7fe4fc7864b91c3ac49a603(size=43.2 M), total size for store is 6.1 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-23 01:31:34,310 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=43.4 M, priority=21, time=3830618776772526; duration=3sec 2015-08-23 01:32:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21205341, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37589, evicted=173, evictedPerRun=0.004602410364896059 2015-08-23 01:34:30,870 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:34:31,657 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98036806, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/f573fc6b4bd94c45880a0199fbfe304c 2015-08-23 01:34:31,710 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/f573fc6b4bd94c45880a0199fbfe304c, entries=321643, sequenceid=98036806, filesize=8.5 M 2015-08-23 01:34:31,711 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219800, currentsize=352.16 KB/360608 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 841ms, sequenceid=98036806, compaction requested=true 2015-08-23 01:35:15,487 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 157 ms, current pipeline: [172.24.32.16:10110, 172.24.32.7:10110, 172.24.32.8:10110] 2015-08-23 01:35:15,537 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293435728 with entries=48474, filesize=121.78 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293715210 2015-08-23 01:35:15,539 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292318012 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292318012 2015-08-23 01:35:15,638 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292596032 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292596032 2015-08-23 01:35:28,526 INFO [sync.2] wal.FSHLog: Slow sync cost: 197 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.7:10110] 2015-08-23 01:35:28,527 INFO [sync.3] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.7:10110] 2015-08-23 01:35:28,527 INFO [sync.0] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.7:10110] 2015-08-23 01:35:28,527 INFO [sync.4] wal.FSHLog: Slow sync cost: 178 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.7:10110] 2015-08-23 01:35:28,527 INFO [sync.1] wal.FSHLog: Slow sync cost: 122 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.7:10110] 2015-08-23 01:37:14,230 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:37:15,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101874199, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/b125dd7803f243f68aec3efa743bb88d 2015-08-23 01:37:15,185 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/b125dd7803f243f68aec3efa743bb88d, entries=366332, sequenceid=101874199, filesize=8.7 M 2015-08-23 01:37:15,186 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221056, currentsize=128.41 KB/131488 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 956ms, sequenceid=101874199, compaction requested=true 2015-08-23 01:37:15,189 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 01:37:15,189 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=43.3 M 2015-08-23 01:37:15,196 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:37:19,275 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 3cb30da13f3b446788c4bbc5ce24973c(size=43.0 M), total size for store is 8.6 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-23 01:37:19,275 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=43.3 M, priority=20, time=3830963600199907; duration=4sec 2015-08-23 01:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21212062, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37619, evicted=173, evictedPerRun=0.0045987400226294994 2015-08-23 01:38:23,269 INFO [sync.0] wal.FSHLog: Slow sync cost: 116 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.7:10110] 2015-08-23 01:39:54,891 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293715210 with entries=48316, filesize=121.61 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293994732 2015-08-23T01:42:02.026+0000: [CMS-concurrent-mark-start] 2015-08-23T01:42:02.114+0000: [CMS-concurrent-mark: 0.088/0.088 secs] [Times: user=0.46 sys=0.00, real=0.09 secs] 2015-08-23T01:42:02.114+0000: [CMS-concurrent-preclean-start] 2015-08-23T01:42:02.118+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T01:42:02.118+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T01:42:05.007+0000: [CMS-concurrent-abortable-preclean: 0.784/2.889 secs] [Times: user=1.02 sys=0.03, real=2.89 secs] 2015-08-23T01:42:05.051+0000: [CMS-concurrent-sweep-start] 2015-08-23T01:42:05.258+0000: [CMS-concurrent-sweep: 0.207/0.207 secs] [Times: user=0.23 sys=0.00, real=0.21 secs] 2015-08-23T01:42:05.258+0000: [CMS-concurrent-reset-start] 2015-08-23T01:42:05.279+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.02 sys=0.01, real=0.02 secs] 2015-08-23 01:42:19,208 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:42:20,048 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100088815, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/3f85bb3ae4d44cdeaabb1baadde69dc1 2015-08-23 01:42:20,092 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/3f85bb3ae4d44cdeaabb1baadde69dc1, entries=384553, sequenceid=100088815, filesize=8.9 M 2015-08-23 01:42:20,094 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134228088, currentsize=146.36 KB/149872 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 886ms, sequenceid=100088815, compaction requested=true 2015-08-23 01:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21212062, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37649, evicted=173, evictedPerRun=0.004595075733959675 2015-08-23 01:44:32,713 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293994732 with entries=48747, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294272526 2015-08-23 01:46:55,597 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.07 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:46:56,361 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98092379, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/d0c280ebb4d04ec08c9fa0c7117ee8e5 2015-08-23 01:46:56,398 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/d0c280ebb4d04ec08c9fa0c7117ee8e5, entries=322314, sequenceid=98092379, filesize=8.6 M 2015-08-23 01:46:56,399 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134286336, currentsize=120.49 KB/123384 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 803ms, sequenceid=98092379, compaction requested=true 2015-08-23 01:46:56,402 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 01:46:56,402 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=25.7 M 2015-08-23 01:46:56,412 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 01:46:57,227 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:46:58,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90829136, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/9d9424e6906d48c4aaa2f7ab0ccc4c9b 2015-08-23 01:46:58,155 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/9d9424e6906d48c4aaa2f7ab0ccc4c9b, entries=367625, sequenceid=90829136, filesize=8.7 M 2015-08-23 01:46:58,156 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134233472, currentsize=123.69 KB/126656 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 929ms, sequenceid=90829136, compaction requested=true 2015-08-23 01:46:58,869 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into b842db5103164ce8a5acab0f7f5f26fe(size=25.4 M), total size for store is 7.3 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 01:46:58,871 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=25.7 M, priority=19, time=3831544813519310; duration=2sec 2015-08-23 01:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21216356, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37679, evicted=173, evictedPerRun=0.004591417033225298 2015-08-23 01:49:13,384 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294272526 with entries=49941, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294553153 2015-08-23 01:50:10,111 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:50:10,973 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101941963, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/6760fa7c933e43e28fb280ad602e847b 2015-08-23 01:50:11,025 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/6760fa7c933e43e28fb280ad602e847b, entries=365844, sequenceid=101941963, filesize=8.7 M 2015-08-23 01:50:11,027 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218368, currentsize=93.59 KB/95840 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 916ms, sequenceid=101941963, compaction requested=true 2015-08-23T01:52:04.166+0000: [CMS-concurrent-mark-start] 2015-08-23T01:52:04.225+0000: [CMS-concurrent-mark: 0.059/0.059 secs] [Times: user=0.30 sys=0.00, real=0.06 secs] 2015-08-23T01:52:04.225+0000: [CMS-concurrent-preclean-start] 2015-08-23T01:52:04.229+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T01:52:04.264+0000: [CMS-concurrent-sweep-start] 2015-08-23T01:52:04.487+0000: [CMS-concurrent-sweep: 0.222/0.223 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-23T01:52:04.488+0000: [CMS-concurrent-reset-start] 2015-08-23T01:52:04.516+0000: [CMS-concurrent-reset: 0.028/0.028 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 2015-08-23 01:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21216356, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37709, evicted=173, evictedPerRun=0.004587764386087656 2015-08-23 01:53:53,312 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 124 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.9:10110] 2015-08-23 01:53:53,363 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294553153 with entries=49745, filesize=121.77 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294833012 2015-08-23 01:56:14,112 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:56:14,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100161148, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/7e5fa57b66ef408497dcb1565f533499 2015-08-23 01:56:15,005 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/7e5fa57b66ef408497dcb1565f533499, entries=384876, sequenceid=100161148, filesize=8.9 M 2015-08-23 01:56:15,006 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220608, currentsize=87.96 KB/90072 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 895ms, sequenceid=100161148, compaction requested=true 2015-08-23 01:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21216356, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37739, evicted=173, evictedPerRun=0.004584117326885462 2015-08-23 01:58:32,465 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.4:10110] 2015-08-23 01:58:32,524 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294833012 with entries=49146, filesize=122.06 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295112040 2015-08-23 01:59:25,852 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 01:59:26,638 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98149679, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/53fa75939fd14dc2803a61a49733d14a 2015-08-23 01:59:26,740 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/53fa75939fd14dc2803a61a49733d14a, entries=322371, sequenceid=98149679, filesize=8.6 M 2015-08-23 01:59:26,741 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218704, currentsize=93.36 KB/95600 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 889ms, sequenceid=98149679, compaction requested=true 2015-08-23 02:02:27,206 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:02:28,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90901941, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/18bd1046ab044ef7aec28a47031516cf 2015-08-23 02:02:28,067 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/18bd1046ab044ef7aec28a47031516cf, entries=367532, sequenceid=90901941, filesize=8.7 M 2015-08-23 02:02:28,068 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221480, currentsize=150.56 KB/154176 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 862ms, sequenceid=90901941, compaction requested=true 2015-08-23 02:02:28,070 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 02:02:28,070 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=60.6 M 2015-08-23 02:02:28,078 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:02:33,356 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 373a54dcb3bc4e63b62a1df6326afc8f(size=60.4 M), total size for store is 6.2 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 02:02:33,356 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=60.6 M, priority=21, time=3832476481407370; duration=5sec 2015-08-23 02:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21225737, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37769, evicted=173, evictedPerRun=0.004580475855618715 2015-08-23 02:03:06,110 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:03:06,951 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102010604, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/90822407cb8d43e6b6716d302f789e54 2015-08-23 02:03:06,987 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/90822407cb8d43e6b6716d302f789e54, entries=365478, sequenceid=102010604, filesize=8.7 M 2015-08-23 02:03:06,989 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223040, currentsize=109.44 KB/112064 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 879ms, sequenceid=102010604, compaction requested=true 2015-08-23 02:03:06,991 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 02:03:06,991 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=60.4 M 2015-08-23 02:03:06,998 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:03:12,252 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 97c2378f6ae6437cac998fae35b6b562(size=60.2 M), total size for store is 8.6 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 02:03:12,252 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=60.4 M, priority=20, time=3832515402671527; duration=5sec 2015-08-23 02:03:13,861 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295112040 with entries=50087, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295393693 2015-08-23 02:04:49,078 INFO [sync.4] wal.FSHLog: Slow sync cost: 114 ms, current pipeline: [172.24.32.16:10110, 172.24.32.7:10110, 172.24.32.13:10110] 2015-08-23 02:04:49,078 INFO [sync.0] wal.FSHLog: Slow sync cost: 109 ms, current pipeline: [172.24.32.16:10110, 172.24.32.7:10110, 172.24.32.13:10110] 2015-08-23T02:05:08.365+0000: [CMS-concurrent-mark-start] 2015-08-23T02:05:08.422+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.29 sys=0.00, real=0.06 secs] 2015-08-23T02:05:08.422+0000: [CMS-concurrent-preclean-start] 2015-08-23T02:05:08.425+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T02:05:08.454+0000: [CMS-concurrent-sweep-start] 2015-08-23T02:05:08.677+0000: [CMS-concurrent-sweep: 0.223/0.223 secs] [Times: user=0.25 sys=0.00, real=0.22 secs] 2015-08-23T02:05:08.677+0000: [CMS-concurrent-reset-start] 2015-08-23T02:05:08.698+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 02:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21235137, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37799, evicted=173, evictedPerRun=0.004576840437948704 2015-08-23 02:07:53,440 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295393693 with entries=50583, filesize=121.77 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295673217 2015-08-23 02:10:10,012 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:10:10,837 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100234699, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/616cb3928ce247448cd4d89739578baa 2015-08-23 02:10:10,881 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/616cb3928ce247448cd4d89739578baa, entries=385021, sequenceid=100234699, filesize=8.9 M 2015-08-23 02:10:10,883 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219656, currentsize=102.64 KB/105104 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 871ms, sequenceid=100234699, compaction requested=true 2015-08-23 02:10:10,885 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 02:10:10,885 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=26.6 M 2015-08-23 02:10:10,896 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:10:13,512 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 8d2bc216e473423c9b30abd4a4bca0e2(size=26.4 M), total size for store is 7.8 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 02:10:13,512 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=26.6 M, priority=19, time=3832939296729926; duration=2sec 2015-08-23 02:11:57,690 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:11:58,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98207408, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/d6648c6212574bed865d53790bf2f621 2015-08-23 02:11:58,560 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/d6648c6212574bed865d53790bf2f621, entries=323581, sequenceid=98207408, filesize=8.6 M 2015-08-23 02:11:58,561 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220056, currentsize=85.49 KB/87544 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 871ms, sequenceid=98207408, compaction requested=true 2015-08-23 02:11:58,564 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 02:11:58,564 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=42.6 M 2015-08-23 02:11:58,572 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:12:02,257 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 28f0f983dc304a189c66addfdc70fa5e(size=42.3 M), total size for store is 7.3 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-23 02:12:02,257 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=42.6 M, priority=19, time=3833046975818843; duration=3sec 2015-08-23 02:12:34,036 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295673217 with entries=51785, filesize=121.71 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295953815 2015-08-23 02:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21246209, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37829, evicted=173, evictedPerRun=0.004573211073875427 2015-08-23 02:16:02,075 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:16:02,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102080860, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/9b5598e8d8904b12bd3480af5ac8968a 2015-08-23 02:16:03,000 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/9b5598e8d8904b12bd3480af5ac8968a, entries=365731, sequenceid=102080860, filesize=8.7 M 2015-08-23 02:16:03,001 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218504, currentsize=125.23 KB/128240 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 926ms, sequenceid=102080860, compaction requested=true 2015-08-23 02:17:14,339 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295953815 with entries=50581, filesize=121.73 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296234091 2015-08-23 02:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21246209, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37859, evicted=173, evictedPerRun=0.004569587297737598 2015-08-23 02:17:58,031 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:17:58,885 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=90976847, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/ab88e049ebae4606bde4fdd3a1fb5591 2015-08-23 02:17:58,937 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/ab88e049ebae4606bde4fdd3a1fb5591, entries=367948, sequenceid=90976847, filesize=8.7 M 2015-08-23 02:17:58,938 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219176, currentsize=115.05 KB/117808 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 907ms, sequenceid=90976847, compaction requested=true 2015-08-23T02:18:30.820+0000: [CMS-concurrent-mark-start] 2015-08-23T02:18:30.880+0000: [CMS-concurrent-mark: 0.060/0.060 secs] [Times: user=0.32 sys=0.00, real=0.06 secs] 2015-08-23T02:18:30.880+0000: [CMS-concurrent-preclean-start] 2015-08-23T02:18:30.885+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T02:18:30.885+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T02:18:33.744+0000: [CMS-concurrent-abortable-preclean: 0.655/2.859 secs] [Times: user=0.88 sys=0.06, real=2.86 secs] 2015-08-23T02:18:33.785+0000: [CMS-concurrent-sweep-start] 2015-08-23T02:18:34.013+0000: [CMS-concurrent-sweep: 0.228/0.228 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T02:18:34.013+0000: [CMS-concurrent-reset-start] 2015-08-23T02:18:34.034+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 02:21:54,134 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.3:10110] 2015-08-23 02:21:54,193 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296234091 with entries=49199, filesize=121.77 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296513946 2015-08-23 02:21:59,325 INFO [sync.0] wal.FSHLog: Slow sync cost: 219 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.13:10110] 2015-08-23 02:21:59,325 INFO [sync.1] wal.FSHLog: Slow sync cost: 214 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.13:10110] 2015-08-23 02:21:59,326 INFO [sync.3] wal.FSHLog: Slow sync cost: 196 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.13:10110] 2015-08-23 02:21:59,326 INFO [sync.2] wal.FSHLog: Slow sync cost: 201 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.13:10110] 2015-08-23 02:21:59,326 INFO [sync.4] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.13:10110] 2015-08-23 02:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21246209, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37889, evicted=173, evictedPerRun=0.004565969109535217 2015-08-23 02:23:20,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 9387 2015-08-23 02:23:30,263 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.26 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:23:30,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57432489, memsize=56.3 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/21ff9c0e3dc3490fb0ef59bab08f5ddc 2015-08-23 02:23:31,030 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/21ff9c0e3dc3490fb0ef59bab08f5ddc, entries=195667, sequenceid=57432489, filesize=3.6 M 2015-08-23 02:23:31,031 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~56.26 MB/58996560, currentsize=6.22 KB/6368 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 769ms, sequenceid=57432489, compaction requested=true 2015-08-23 02:23:31,033 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. 2015-08-23 02:23:31,033 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp, totalSize=79.9 M 2015-08-23 02:23:31,038 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:23:38,703 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into e3243f1323374780ba01bc64db12a648(size=79.9 M), total size for store is 1.8 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-23 02:23:38,703 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., storeName=e, fileCount=3, fileSize=79.9 M, priority=20, time=3833739444138437; duration=7sec 2015-08-23 02:24:09,134 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:24:10,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100308045, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/123742a47bda4588a8ee7e06681c49eb 2015-08-23 02:24:10,091 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/123742a47bda4588a8ee7e06681c49eb, entries=384798, sequenceid=100308045, filesize=8.9 M 2015-08-23 02:24:10,092 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134240952, currentsize=153.32 KB/157000 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 958ms, sequenceid=100308045, compaction requested=true 2015-08-23 02:24:25,116 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:24:25,916 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98263660, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/acf94692ac2148bba7c3f2eb768c8452 2015-08-23 02:24:25,976 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/acf94692ac2148bba7c3f2eb768c8452, entries=321567, sequenceid=98263660, filesize=8.6 M 2015-08-23 02:24:25,977 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134227000, currentsize=224.24 KB/229624 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 861ms, sequenceid=98263660, compaction requested=true 2015-08-23 02:26:33,470 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.13:10110] 2015-08-23 02:26:33,521 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296513946 with entries=48890, filesize=121.76 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296793228 2015-08-23 02:26:33,523 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292874930 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440292874930 2015-08-23 02:26:33,613 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293156250 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293156250 2015-08-23 02:26:33,696 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293435728 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293435728 2015-08-23 02:26:33,779 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293715210 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293715210 2015-08-23 02:26:33,863 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293994732 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440293994732 2015-08-23 02:26:33,946 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294272526 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294272526 2015-08-23 02:26:34,029 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294553153 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294553153 2015-08-23 02:26:34,113 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294833012 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440294833012 2015-08-23 02:26:34,196 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295112040 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295112040 2015-08-23 02:26:34,280 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295393693 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295393693 2015-08-23 02:26:34,363 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295673217 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295673217 2015-08-23 02:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21257875, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37919, evicted=173, evictedPerRun=0.004562356509268284 2015-08-23 02:28:57,690 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:28:58,552 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102148765, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/7ef44e6cd7ac4b6abf04411d337369bd 2015-08-23 02:28:58,605 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/7ef44e6cd7ac4b6abf04411d337369bd, entries=365945, sequenceid=102148765, filesize=8.7 M 2015-08-23 02:28:58,607 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221152, currentsize=127.45 KB/130504 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 917ms, sequenceid=102148765, compaction requested=true 2015-08-23 02:28:58,609 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 02:28:58,610 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=77.6 M 2015-08-23 02:28:58,615 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:29:05,295 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into d262ec1cec864ed9b79b275606d2716b(size=77.4 M), total size for store is 8.6 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 02:29:05,295 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=77.6 M, priority=20, time=3834067020883173; duration=6sec 2015-08-23 02:31:13,141 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296793228 with entries=49364, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297072913 2015-08-23 02:31:13,144 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295953815 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440295953815 2015-08-23 02:31:31,950 INFO [sync.2] wal.FSHLog: Slow sync cost: 160 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:31:31,950 INFO [sync.3] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:31:31,950 INFO [sync.4] wal.FSHLog: Slow sync cost: 141 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:31:31,952 INFO [sync.0] wal.FSHLog: Slow sync cost: 127 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23T02:31:45.803+0000: [CMS-concurrent-mark-start] 2015-08-23T02:31:45.874+0000: [CMS-concurrent-mark: 0.071/0.071 secs] [Times: user=0.36 sys=0.00, real=0.07 secs] 2015-08-23T02:31:45.874+0000: [CMS-concurrent-preclean-start] 2015-08-23T02:31:45.878+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T02:31:45.878+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T02:31:48.745+0000: [CMS-concurrent-abortable-preclean: 0.763/2.867 secs] [Times: user=0.99 sys=0.05, real=2.87 secs] 2015-08-23T02:31:48.789+0000: [CMS-concurrent-sweep-start] 2015-08-23T02:31:49.003+0000: [CMS-concurrent-sweep: 0.215/0.215 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-23T02:31:49.004+0000: [CMS-concurrent-reset-start] 2015-08-23T02:31:49.025+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 02:32:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21269954, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37949, evicted=173, evictedPerRun=0.004558749962598085 2015-08-23 02:33:14,798 INFO [sync.1] wal.FSHLog: Slow sync cost: 134 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:33:29,431 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:33:30,284 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91049342, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/afdba0716c6a4fc6bdd0aeaaf235d647 2015-08-23 02:33:30,336 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/afdba0716c6a4fc6bdd0aeaaf235d647, entries=367698, sequenceid=91049342, filesize=8.7 M 2015-08-23 02:33:30,337 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222320, currentsize=108.96 KB/111576 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 906ms, sequenceid=91049342, compaction requested=true 2015-08-23 02:33:30,338 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 02:33:30,338 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=77.8 M 2015-08-23 02:33:30,343 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:33:37,201 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 64ce992bbc094002a1bbee0ead2b1799(size=77.7 M), total size for store is 6.2 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 02:33:37,201 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=77.8 M, priority=21, time=3834338749656060; duration=6sec 2015-08-23 02:35:02,680 INFO [sync.0] wal.FSHLog: Slow sync cost: 116 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:35:02,680 INFO [sync.4] wal.FSHLog: Slow sync cost: 128 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:35:52,157 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 02:35:52,208 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297072913 with entries=49713, filesize=121.72 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297351907 2015-08-23 02:35:52,210 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296234091 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296234091 2015-08-23 02:36:51,568 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:36:52,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98319547, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/0b992d44341a4f258492084e6ae63761 2015-08-23 02:36:52,408 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/0b992d44341a4f258492084e6ae63761, entries=321331, sequenceid=98319547, filesize=8.6 M 2015-08-23 02:36:52,409 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217784, currentsize=190.19 KB/194752 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 841ms, sequenceid=98319547, compaction requested=true 2015-08-23 02:36:52,412 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 02:36:52,412 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=59.5 M 2015-08-23 02:36:52,419 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:36:57,723 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into babb43b0a17c41bd93303cbaf6d813c5(size=59.2 M), total size for store is 7.3 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 02:36:57,723 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=59.5 M, priority=19, time=3834540823468610; duration=5sec 2015-08-23 02:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21291981, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=37979, evicted=173, evictedPerRun=0.004555149003863335 2015-08-23 02:38:08,066 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:38:08,938 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100380944, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/0642e48f84b94ebb84a9bcee473c20df 2015-08-23 02:38:08,991 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/0642e48f84b94ebb84a9bcee473c20df, entries=384855, sequenceid=100380944, filesize=8.9 M 2015-08-23 02:38:08,993 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219712, currentsize=104.05 KB/106544 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 926ms, sequenceid=100380944, compaction requested=true 2015-08-23 02:38:08,995 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 02:38:08,995 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=44.2 M 2015-08-23 02:38:09,001 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:38:13,114 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 12f14cf8c7374688a13a516e42a2e5af(size=43.9 M), total size for store is 7.8 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-23 02:38:13,114 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=44.2 M, priority=19, time=3834617406144378; duration=4sec 2015-08-23 02:40:30,237 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.14:10110] 2015-08-23 02:40:30,296 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297351907 with entries=50531, filesize=121.70 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297630032 2015-08-23 02:40:35,408 INFO [sync.2] wal.FSHLog: Slow sync cost: 203 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.8:10110] 2015-08-23 02:40:35,408 INFO [sync.4] wal.FSHLog: Slow sync cost: 179 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.8:10110] 2015-08-23 02:40:35,408 INFO [sync.0] wal.FSHLog: Slow sync cost: 164 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.8:10110] 2015-08-23 02:40:35,408 INFO [sync.3] wal.FSHLog: Slow sync cost: 183 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.8:10110] 2015-08-23 02:40:35,408 INFO [sync.1] wal.FSHLog: Slow sync cost: 139 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.8:10110] 2015-08-23 02:41:52,614 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:41:53,471 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102217141, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/3d55fbbbe6ed409ab2a755765119972a 2015-08-23 02:41:53,532 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/3d55fbbbe6ed409ab2a755765119972a, entries=365287, sequenceid=102217141, filesize=8.7 M 2015-08-23 02:41:53,532 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218208, currentsize=144.35 KB/147816 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 918ms, sequenceid=102217141, compaction requested=true 2015-08-23 02:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21298524, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38009, evicted=173, evictedPerRun=0.004551553633064032 2015-08-23T02:43:15.577+0000: [CMS-concurrent-mark-start] 2015-08-23T02:43:15.634+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.29 sys=0.00, real=0.06 secs] 2015-08-23T02:43:15.635+0000: [CMS-concurrent-preclean-start] 2015-08-23T02:43:15.637+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T02:43:15.669+0000: [CMS-concurrent-sweep-start] 2015-08-23T02:43:15.897+0000: [CMS-concurrent-sweep: 0.227/0.227 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T02:43:15.897+0000: [CMS-concurrent-reset-start] 2015-08-23T02:43:15.918+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 02:45:11,533 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 166 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.8:10110] 2015-08-23 02:45:11,592 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297630032 with entries=49919, filesize=121.75 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297911251 2015-08-23 02:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21298524, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38039, evicted=173, evictedPerRun=0.004547963850200176 2015-08-23 02:48:59,010 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:48:59,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91124264, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/653cbcb3c99d41f69827bf58200b812b 2015-08-23 02:48:59,911 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/653cbcb3c99d41f69827bf58200b812b, entries=368020, sequenceid=91124264, filesize=8.7 M 2015-08-23 02:48:59,913 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218160, currentsize=146.51 KB/150024 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 904ms, sequenceid=91124264, compaction requested=true 2015-08-23 02:49:19,589 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:49:20,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98376691, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/91922286489b4631ab6afeb5364ad741 2015-08-23 02:49:20,454 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/91922286489b4631ab6afeb5364ad741, entries=322197, sequenceid=98376691, filesize=8.6 M 2015-08-23 02:49:20,455 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218752, currentsize=117.35 KB/120168 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 866ms, sequenceid=98376691, compaction requested=true 2015-08-23 02:49:52,472 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297911251 with entries=50083, filesize=121.74 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298192214 2015-08-23 02:52:06,845 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:52:07,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100454314, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/f1e3629c7f2744f5963804a1665715a9 2015-08-23 02:52:07,772 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/f1e3629c7f2744f5963804a1665715a9, entries=384927, sequenceid=100454314, filesize=8.9 M 2015-08-23 02:52:07,774 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217800, currentsize=174.88 KB/179080 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 928ms, sequenceid=100454314, compaction requested=true 2015-08-23 02:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21298524, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38069, evicted=173, evictedPerRun=0.004544380120933056 2015-08-23 02:53:28,321 INFO [sync.2] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.11:10110] 2015-08-23 02:54:30,352 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298192214 with entries=49316, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298470172 2015-08-23 02:54:50,034 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 02:54:50,838 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102285300, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/27b03ad5317646b89a8120b4942c132f 2015-08-23 02:54:50,882 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/27b03ad5317646b89a8120b4942c132f, entries=365334, sequenceid=102285300, filesize=8.7 M 2015-08-23 02:54:50,883 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219888, currentsize=108.70 KB/111304 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 849ms, sequenceid=102285300, compaction requested=true 2015-08-23 02:54:50,886 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 02:54:50,886 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=94.8 M 2015-08-23 02:54:50,909 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 02:54:58,880 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 12b4d9d56312442f87acdd0dcc883396(size=94.6 M), total size for store is 8.6 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-23 02:54:58,880 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=94.8 M, priority=20, time=3835619297076499; duration=7sec 2015-08-23T02:56:28.712+0000: [CMS-concurrent-mark-start] 2015-08-23T02:56:28.769+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.30 sys=0.00, real=0.05 secs] 2015-08-23T02:56:28.769+0000: [CMS-concurrent-preclean-start] 2015-08-23T02:56:28.773+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T02:56:28.773+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T02:56:31.685+0000: [CMS-concurrent-abortable-preclean: 0.608/2.912 secs] [Times: user=0.86 sys=0.04, real=2.91 secs] 2015-08-23T02:56:31.723+0000: [CMS-concurrent-sweep-start] 2015-08-23T02:56:31.954+0000: [CMS-concurrent-sweep: 0.231/0.231 secs] [Times: user=0.26 sys=0.00, real=0.23 secs] 2015-08-23T02:56:31.954+0000: [CMS-concurrent-reset-start] 2015-08-23T02:56:31.975+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 02:56:58,767 INFO [sync.4] wal.FSHLog: Slow sync cost: 183 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.8:10110] 2015-08-23 02:56:58,767 INFO [sync.3] wal.FSHLog: Slow sync cost: 200 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.8:10110] 2015-08-23 02:56:58,768 INFO [sync.0] wal.FSHLog: Slow sync cost: 163 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.8:10110] 2015-08-23 02:56:58,768 INFO [sync.1] wal.FSHLog: Slow sync cost: 159 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.8:10110] 2015-08-23 02:56:58,768 INFO [sync.2] wal.FSHLog: Slow sync cost: 144 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.8:10110] 2015-08-23 02:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21313284, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38099, evicted=173, evictedPerRun=0.004540801513940096 2015-08-23 02:59:11,416 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298470172 with entries=49815, filesize=121.65 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298751234 2015-08-23 03:01:48,932 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:01:49,725 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98433276, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/9b0992d58c0344e5b813cb2d56a49120 2015-08-23 03:01:49,769 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/9b0992d58c0344e5b813cb2d56a49120, entries=321913, sequenceid=98433276, filesize=8.6 M 2015-08-23 03:01:49,771 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217952, currentsize=146.51 KB/150024 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 839ms, sequenceid=98433276, compaction requested=true 2015-08-23 03:01:49,773 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 03:01:49,773 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=76.3 M 2015-08-23 03:01:49,781 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:01:56,334 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 3edc378be6ca4f519b8e40eeeb965552(size=76.1 M), total size for store is 7.3 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 03:01:56,334 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=76.3 M, priority=19, time=3836038184374016; duration=6sec 2015-08-23 03:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21326099, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38129, evicted=173, evictedPerRun=0.004537228960543871 2015-08-23 03:03:51,230 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298751234 with entries=49789, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299031068 2015-08-23 03:04:33,411 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:04:34,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91198429, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/aa6b1c6adb784ae68858a3d806986908 2015-08-23 03:04:34,263 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/aa6b1c6adb784ae68858a3d806986908, entries=367219, sequenceid=91198429, filesize=8.7 M 2015-08-23 03:04:34,264 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222848, currentsize=103.34 KB/105816 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 853ms, sequenceid=91198429, compaction requested=true 2015-08-23 03:04:34,266 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 03:04:34,266 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=95.1 M 2015-08-23 03:04:34,273 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:04:42,202 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into c0c64becea3d4e0892c410eb80ca40c4(size=95.0 M), total size for store is 6.2 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-23 03:04:42,202 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=95.1 M, priority=21, time=3836202677456491; duration=7sec 2015-08-23 03:06:04,151 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:06:04,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100527339, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/ae26d64a4ffc484099df1a3a391bf259 2015-08-23 03:06:05,021 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/ae26d64a4ffc484099df1a3a391bf259, entries=384885, sequenceid=100527339, filesize=8.9 M 2015-08-23 03:06:05,022 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218920, currentsize=77.04 KB/78888 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 871ms, sequenceid=100527339, compaction requested=true 2015-08-23 03:06:05,024 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 03:06:05,024 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 5 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=387.4 M 2015-08-23 03:06:05,033 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:06:37,489 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 5 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 91f805d335a24e90b229e9c96cdefbbe(size=387.8 M), total size for store is 7.9 G. This selection was in queue for 0sec, and took 32sec to execute. 2015-08-23 03:06:37,489 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=5, fileSize=387.4 M, priority=19, time=3836293435714109; duration=32sec 2015-08-23 03:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21398162, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38159, evicted=173, evictedPerRun=0.004533661995083094 2015-08-23 03:07:50,634 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:07:51,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102354506, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/03b33863d7594f448428b254dd249d50 2015-08-23 03:07:51,491 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/03b33863d7594f448428b254dd249d50, entries=367006, sequenceid=102354506, filesize=8.7 M 2015-08-23 03:07:51,492 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218856, currentsize=168.84 KB/172888 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 858ms, sequenceid=102354506, compaction requested=true 2015-08-23 03:08:31,318 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299031068 with entries=50750, filesize=121.72 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299311114 2015-08-23T03:09:41.419+0000: [CMS-concurrent-mark-start] 2015-08-23T03:09:41.477+0000: [CMS-concurrent-mark: 0.058/0.058 secs] [Times: user=0.30 sys=0.00, real=0.06 secs] 2015-08-23T03:09:41.477+0000: [CMS-concurrent-preclean-start] 2015-08-23T03:09:41.481+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-23T03:09:41.481+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T03:09:44.476+0000: [CMS-concurrent-abortable-preclean: 0.792/2.996 secs] [Times: user=1.03 sys=0.04, real=3.00 secs] 2015-08-23T03:09:44.519+0000: [CMS-concurrent-sweep-start] 2015-08-23T03:09:44.739+0000: [CMS-concurrent-sweep: 0.220/0.220 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-23T03:09:44.739+0000: [CMS-concurrent-reset-start] 2015-08-23T03:09:44.760+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 03:10:34,306 INFO [sync.4] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 03:10:34,306 INFO [sync.3] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 03:10:34,306 INFO [sync.2] wal.FSHLog: Slow sync cost: 160 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 03:10:34,306 INFO [sync.0] wal.FSHLog: Slow sync cost: 117 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.8:10110] 2015-08-23 03:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21398162, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38189, evicted=173, evictedPerRun=0.004530100151896477 2015-08-23 03:13:11,256 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299311114 with entries=49587, filesize=121.76 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299591065 2015-08-23 03:14:21,208 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:14:22,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98489968, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/320655722a64485390ce06ee015fdecc 2015-08-23 03:14:22,065 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/320655722a64485390ce06ee015fdecc, entries=323100, sequenceid=98489968, filesize=8.6 M 2015-08-23 03:14:22,066 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134224584, currentsize=291.71 KB/298712 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 858ms, sequenceid=98489968, compaction requested=true 2015-08-23 03:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21398162, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38219, evicted=173, evictedPerRun=0.004526544362306595 2015-08-23 03:17:49,475 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299591065 with entries=48548, filesize=121.73 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299869305 2015-08-23 03:19:59,905 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:20:00,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100599547, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/3d0659e3a7584b9a954e9428e4aa5371 2015-08-23 03:20:00,732 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/3d0659e3a7584b9a954e9428e4aa5371, entries=384866, sequenceid=100599547, filesize=8.8 M 2015-08-23 03:20:00,734 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218608, currentsize=106.69 KB/109248 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 828ms, sequenceid=100599547, compaction requested=true 2015-08-23 03:20:03,433 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:20:04,206 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91272885, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/c3a20030f829414a8eb1b70adeddbc66 2015-08-23 03:20:04,226 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/c3a20030f829414a8eb1b70adeddbc66, entries=366961, sequenceid=91272885, filesize=8.7 M 2015-08-23 03:20:04,227 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221424, currentsize=107.17 KB/109744 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 794ms, sequenceid=91272885, compaction requested=true 2015-08-23 03:20:41,210 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:20:41,985 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102421327, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/c5025196e7434d4cbe158f57034da9df 2015-08-23 03:20:42,005 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/c5025196e7434d4cbe158f57034da9df, entries=364064, sequenceid=102421327, filesize=8.7 M 2015-08-23 03:20:42,006 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222576, currentsize=144.63 KB/148104 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 796ms, sequenceid=102421327, compaction requested=true 2015-08-23 03:20:42,008 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 03:20:42,008 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=112.0 M 2015-08-23 03:20:42,017 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:20:51,244 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into aeac76ee834e4dff951a8056b6b88cbb(size=111.8 M), total size for store is 8.6 G. This selection was in queue for 0sec, and took 9sec to execute. 2015-08-23 03:20:51,244 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=112.0 M, priority=20, time=3837170419708364; duration=9sec 2015-08-23 03:22:30,222 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299869305 with entries=48790, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300150127 2015-08-23 03:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21415601, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38249, evicted=173, evictedPerRun=0.004522994160652161 2015-08-23T03:23:09.442+0000: [CMS-concurrent-mark-start] 2015-08-23T03:23:09.499+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.30 sys=0.00, real=0.05 secs] 2015-08-23T03:23:09.500+0000: [CMS-concurrent-preclean-start] 2015-08-23T03:23:09.503+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T03:23:09.503+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T03:23:12.712+0000: [CMS-concurrent-abortable-preclean: 0.806/3.209 secs] [Times: user=1.03 sys=0.04, real=3.21 secs] 2015-08-23T03:23:12.747+0000: [CMS-concurrent-sweep-start] 2015-08-23T03:23:12.973+0000: [CMS-concurrent-sweep: 0.225/0.225 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T03:23:12.973+0000: [CMS-concurrent-reset-start] 2015-08-23T03:23:12.994+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 03:23:30,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 4590 2015-08-23 03:23:35,465 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.51 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:23:35,879 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57524873, memsize=56.5 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/49c99deb5c69498ea9fd9d9974e28e6f 2015-08-23 03:23:35,906 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/49c99deb5c69498ea9fd9d9974e28e6f, entries=196214, sequenceid=57524873, filesize=3.6 M 2015-08-23 03:23:35,907 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~56.51 MB/59250448, currentsize=5.77 KB/5904 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 442ms, sequenceid=57524873, compaction requested=true 2015-08-23 03:26:49,514 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:26:50,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98545355, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/f5a3e4bfa5e34c37a169b68ab98eddf4 2015-08-23 03:26:50,260 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/f5a3e4bfa5e34c37a169b68ab98eddf4, entries=321251, sequenceid=98545355, filesize=8.6 M 2015-08-23 03:26:50,261 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218536, currentsize=116.35 KB/119144 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 748ms, sequenceid=98545355, compaction requested=true 2015-08-23 03:26:50,263 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 03:26:50,263 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=93.2 M 2015-08-23 03:26:50,271 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:26:57,791 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 24ebe4ec901a45a9a1d20738978b6884(size=93.0 M), total size for store is 7.3 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-23 03:26:57,791 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=93.2 M, priority=19, time=3837538674696225; duration=7sec 2015-08-23 03:27:11,185 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300150127 with entries=49537, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300431135 2015-08-23 03:27:11,187 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296513946 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296513946 2015-08-23 03:27:11,201 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296793228 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440296793228 2015-08-23 03:27:11,218 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297072913 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297072913 2015-08-23 03:27:11,234 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297351907 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297351907 2015-08-23 03:27:11,251 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297630032 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297630032 2015-08-23 03:27:11,268 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297911251 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440297911251 2015-08-23 03:27:11,285 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298192214 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298192214 2015-08-23 03:27:11,301 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298470172 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298470172 2015-08-23 03:27:11,318 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298751234 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440298751234 2015-08-23 03:27:11,334 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299031068 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299031068 2015-08-23 03:27:11,351 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299311114 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299311114 2015-08-23 03:27:11,368 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299591065 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299591065 2015-08-23 03:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21431260, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38279, evicted=173, evictedPerRun=0.004519449081271887 2015-08-23 03:31:50,354 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300431135 with entries=50226, filesize=121.73 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300710269 2015-08-23 03:32:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21431260, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38309, evicted=173, evictedPerRun=0.004515910055488348 2015-08-23 03:33:35,232 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:33:36,048 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102489587, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/4cab525745164900b9a22e78f8a6a53a 2015-08-23 03:33:36,084 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/4cab525745164900b9a22e78f8a6a53a, entries=365101, sequenceid=102489587, filesize=8.7 M 2015-08-23 03:33:36,085 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220568, currentsize=124.86 KB/127856 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 854ms, sequenceid=102489587, compaction requested=true 2015-08-23 03:33:56,306 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:33:57,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100672356, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/55ef8d667af5412cb5e75d581cbdcd79 2015-08-23 03:33:57,153 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/55ef8d667af5412cb5e75d581cbdcd79, entries=384370, sequenceid=100672356, filesize=8.9 M 2015-08-23 03:33:57,154 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217840, currentsize=180.12 KB/184440 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 848ms, sequenceid=100672356, compaction requested=true 2015-08-23 03:35:37,247 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:35:38,053 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91346812, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/e9a41e1f531f47e38d121979a3f0977c 2015-08-23 03:35:38,089 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/e9a41e1f531f47e38d121979a3f0977c, entries=366642, sequenceid=91346812, filesize=8.7 M 2015-08-23 03:35:38,090 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134226768, currentsize=143.60 KB/147048 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 843ms, sequenceid=91346812, compaction requested=true 2015-08-23 03:35:38,092 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 03:35:38,092 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=112.4 M 2015-08-23 03:35:38,100 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:35:47,804 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 9ebfe60170964fe3bdf420fbb4a342df(size=112.2 M), total size for store is 6.2 G. This selection was in queue for 0sec, and took 9sec to execute. 2015-08-23 03:35:47,804 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=112.4 M, priority=21, time=3838066503359184; duration=9sec 2015-08-23T03:36:23.447+0000: [CMS-concurrent-mark-start] 2015-08-23T03:36:23.490+0000: [CMS-concurrent-mark: 0.043/0.043 secs] [Times: user=0.22 sys=0.00, real=0.04 secs] 2015-08-23T03:36:23.490+0000: [CMS-concurrent-preclean-start] 2015-08-23T03:36:23.494+0000: [CMS-concurrent-preclean: 0.003/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T03:36:23.523+0000: [CMS-concurrent-sweep-start] 2015-08-23T03:36:23.754+0000: [CMS-concurrent-sweep: 0.230/0.230 secs] [Times: user=0.26 sys=0.00, real=0.23 secs] 2015-08-23T03:36:23.754+0000: [CMS-concurrent-reset-start] 2015-08-23T03:36:23.774+0000: [CMS-concurrent-reset: 0.020/0.020 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 03:36:28,384 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300710269 with entries=50583, filesize=122.05 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300988213 2015-08-23 03:36:28,386 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299869305 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440299869305 2015-08-23 03:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21448664, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38339, evicted=173, evictedPerRun=0.004512376617640257 2015-08-23 03:39:18,332 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:39:19,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98602653, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/417650f23eb74c32b0ab6ec0d47b8f65 2015-08-23 03:39:19,174 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/417650f23eb74c32b0ab6ec0d47b8f65, entries=322736, sequenceid=98602653, filesize=8.6 M 2015-08-23 03:39:19,176 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217808, currentsize=82.73 KB/84712 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 844ms, sequenceid=98602653, compaction requested=true 2015-08-23 03:41:07,855 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300988213 with entries=50323, filesize=121.61 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301267772 2015-08-23 03:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21448664, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38369, evicted=173, evictedPerRun=0.004508848302066326 2015-08-23 03:45:47,936 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301267772 with entries=49766, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301547805 2015-08-23 03:45:53,104 INFO [sync.3] wal.FSHLog: Slow sync cost: 159 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.11:10110] 2015-08-23 03:45:53,104 INFO [sync.2] wal.FSHLog: Slow sync cost: 174 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.11:10110] 2015-08-23 03:45:53,104 INFO [sync.1] wal.FSHLog: Slow sync cost: 195 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.11:10110] 2015-08-23 03:45:53,104 INFO [sync.0] wal.FSHLog: Slow sync cost: 219 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.11:10110] 2015-08-23 03:45:53,104 INFO [sync.4] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.11:10110] 2015-08-23 03:46:29,274 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.06 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:46:30,073 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102558572, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/670f35516e2b45429790d1eb9a315c0a 2015-08-23 03:46:30,118 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/670f35516e2b45429790d1eb9a315c0a, entries=365625, sequenceid=102558572, filesize=8.7 M 2015-08-23 03:46:30,119 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134278688, currentsize=153.91 KB/157600 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 845ms, sequenceid=102558572, compaction requested=true 2015-08-23 03:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21448664, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38399, evicted=173, evictedPerRun=0.004505325574427843 2015-08-23 03:47:50,926 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:47:51,724 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100746125, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/e42ad6931d424d3ab0684876650fa1be 2015-08-23 03:47:51,760 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/e42ad6931d424d3ab0684876650fa1be, entries=384811, sequenceid=100746125, filesize=8.9 M 2015-08-23 03:47:51,761 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218112, currentsize=171.69 KB/175808 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 835ms, sequenceid=100746125, compaction requested=true 2015-08-23 03:47:51,763 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 03:47:51,763 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=26.6 M 2015-08-23 03:47:51,771 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:47:54,332 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into de79d490beee4c558ab203ca001e8773(size=26.4 M), total size for store is 7.9 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 03:47:54,333 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=26.6 M, priority=20, time=3838800174433194; duration=2sec 2015-08-23 03:50:28,450 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301547805 with entries=49548, filesize=121.89 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301828275 2015-08-23T03:51:07.541+0000: [CMS-concurrent-mark-start] 2015-08-23T03:51:07.624+0000: [CMS-concurrent-mark: 0.083/0.083 secs] [Times: user=0.43 sys=0.00, real=0.09 secs] 2015-08-23T03:51:07.624+0000: [CMS-concurrent-preclean-start] 2015-08-23T03:51:07.628+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T03:51:07.628+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23 03:51:08,066 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23T03:51:08.515+0000: [CMS-concurrent-abortable-preclean: 0.297/0.887 secs] [Times: user=0.75 sys=0.00, real=0.89 secs] 2015-08-23T03:51:08.545+0000: [CMS-concurrent-sweep-start] 2015-08-23T03:51:08.756+0000: [CMS-concurrent-sweep: 0.211/0.211 secs] [Times: user=0.44 sys=0.00, real=0.21 secs] 2015-08-23T03:51:08.756+0000: [CMS-concurrent-reset-start] 2015-08-23T03:51:08.778+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 2015-08-23 03:51:08,904 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91420696, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/1437a5d284124290bc76c2cdde70bb5d 2015-08-23 03:51:08,949 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/1437a5d284124290bc76c2cdde70bb5d, entries=367472, sequenceid=91420696, filesize=8.7 M 2015-08-23 03:51:08,950 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219448, currentsize=56.15 KB/57496 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 884ms, sequenceid=91420696, compaction requested=true 2015-08-23 03:51:47,261 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.10 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:51:48,016 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98658446, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/3dc667e78ae24b13b28689beae49b868 2015-08-23 03:51:48,061 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/3dc667e78ae24b13b28689beae49b868, entries=321714, sequenceid=98658446, filesize=8.6 M 2015-08-23 03:51:48,062 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.10 MB/134324128, currentsize=105.24 KB/107768 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 801ms, sequenceid=98658446, compaction requested=true 2015-08-23 03:51:48,064 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 03:51:48,064 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 4 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=237.1 M 2015-08-23 03:51:48,072 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:52:06,477 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 4 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into f157c4ee06d4497197e897d5a0d95acf(size=237.0 M), total size for store is 7.4 G. This selection was in queue for 0sec, and took 18sec to execute. 2015-08-23 03:52:06,477 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=4, fileSize=237.1 M, priority=19, time=3839036475874582; duration=18sec 2015-08-23 03:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21492434, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38429, evicted=173, evictedPerRun=0.004501808434724808 2015-08-23 03:55:09,713 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301828275 with entries=49615, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302109610 2015-08-23 03:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21492434, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38459, evicted=173, evictedPerRun=0.00449829688295722 2015-08-23 03:59:30,394 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.20 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 03:59:31,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102627383, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/3c9e730566ab484d9b625efcd150711a 2015-08-23 03:59:31,151 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/3c9e730566ab484d9b625efcd150711a, entries=366746, sequenceid=102627383, filesize=8.7 M 2015-08-23 03:59:31,152 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.20 MB/134424248, currentsize=508.98 KB/521200 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 758ms, sequenceid=102627383, compaction requested=true 2015-08-23 03:59:31,153 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 03:59:31,154 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=26.1 M 2015-08-23 03:59:31,161 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 03:59:33,499 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into a1b9249a4e4f463d884bd8d32cc327e8(size=25.9 M), total size for store is 8.7 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 03:59:33,499 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=26.1 M, priority=19, time=3839499565075170; duration=2sec 2015-08-23 03:59:48,418 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302109610 with entries=50146, filesize=121.60 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302388331 2015-08-23T04:01:19.226+0000: [CMS-concurrent-mark-start] 2015-08-23T04:01:19.315+0000: [CMS-concurrent-mark: 0.089/0.089 secs] [Times: user=0.46 sys=0.00, real=0.09 secs] 2015-08-23T04:01:19.315+0000: [CMS-concurrent-preclean-start] 2015-08-23T04:01:19.319+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T04:01:19.319+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T04:01:22.340+0000: [CMS-concurrent-abortable-preclean: 0.817/3.021 secs] [Times: user=1.05 sys=0.04, real=3.02 secs] 2015-08-23T04:01:22.372+0000: [CMS-concurrent-sweep-start] 2015-08-23T04:01:22.573+0000: [CMS-concurrent-sweep: 0.200/0.200 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] 2015-08-23T04:01:22.573+0000: [CMS-concurrent-reset-start] 2015-08-23T04:01:22.594+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 04:01:48,193 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:01:48,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100819523, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/fea8c2c475b1480c82d6b35c45a3256f 2015-08-23 04:01:48,990 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/fea8c2c475b1480c82d6b35c45a3256f, entries=385081, sequenceid=100819523, filesize=8.9 M 2015-08-23 04:01:48,991 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222168, currentsize=68.92 KB/70576 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 798ms, sequenceid=100819523, compaction requested=true 2015-08-23 04:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21496490, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38489, evicted=173, evictedPerRun=0.00449479091912508 2015-08-23 04:04:21,012 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:04:21,737 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98716184, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/6b86f2c36e0446be9ed9290250ebdcd6 2015-08-23 04:04:21,755 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/6b86f2c36e0446be9ed9290250ebdcd6, entries=323560, sequenceid=98716184, filesize=8.6 M 2015-08-23 04:04:21,756 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217808, currentsize=132.80 KB/135984 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 744ms, sequenceid=98716184, compaction requested=true 2015-08-23 04:04:29,305 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302388331 with entries=51001, filesize=121.81 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302669209 2015-08-23 04:06:43,347 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:06:44,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91495351, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/5268142f93574fe2be9fa9e581fabfe4 2015-08-23 04:06:44,230 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/5268142f93574fe2be9fa9e581fabfe4, entries=367259, sequenceid=91495351, filesize=8.7 M 2015-08-23 04:06:44,231 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222176, currentsize=223.62 KB/228984 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 884ms, sequenceid=91495351, compaction requested=true 2015-08-23 04:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21496490, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38519, evicted=173, evictedPerRun=0.0044912900775671005 2015-08-23 04:09:07,667 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302669209 with entries=49571, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302947455 2015-08-23T04:11:24.358+0000: [CMS-concurrent-mark-start] 2015-08-23T04:11:24.446+0000: [CMS-concurrent-mark: 0.088/0.088 secs] [Times: user=0.46 sys=0.00, real=0.09 secs] 2015-08-23T04:11:24.446+0000: [CMS-concurrent-preclean-start] 2015-08-23T04:11:24.450+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T04:11:24.450+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T04:11:27.350+0000: [CMS-concurrent-abortable-preclean: 0.696/2.900 secs] [Times: user=0.94 sys=0.04, real=2.90 secs] 2015-08-23T04:11:27.386+0000: [CMS-concurrent-sweep-start] 2015-08-23T04:11:27.591+0000: [CMS-concurrent-sweep: 0.205/0.205 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] 2015-08-23T04:11:27.591+0000: [CMS-concurrent-reset-start] 2015-08-23T04:11:27.612+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 2015-08-23 04:12:29,915 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.07 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:12:30,763 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102696641, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/cdd6e6b1a649492091d5ff0d8314fb2e 2015-08-23 04:12:30,866 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/cdd6e6b1a649492091d5ff0d8314fb2e, entries=366194, sequenceid=102696641, filesize=8.7 M 2015-08-23 04:12:30,868 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134292400, currentsize=207.49 KB/212472 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 953ms, sequenceid=102696641, compaction requested=true 2015-08-23 04:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21496490, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38549, evicted=173, evictedPerRun=0.004487794823944569 2015-08-23 04:13:47,604 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302947455 with entries=50475, filesize=121.73 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303227331 2015-08-23 04:15:44,507 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:15:45,350 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100893440, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/fb0dce937d8f48ea9b30e29b613b8432 2015-08-23 04:15:45,402 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/fb0dce937d8f48ea9b30e29b613b8432, entries=384752, sequenceid=100893440, filesize=8.9 M 2015-08-23 04:15:45,404 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218296, currentsize=143.05 KB/146480 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 897ms, sequenceid=100893440, compaction requested=true 2015-08-23 04:15:45,406 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 04:15:45,406 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=44.1 M 2015-08-23 04:15:45,414 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:15:49,492 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 56b0f08cb2f64dc4be1662bb584f3e7b(size=43.9 M), total size for store is 7.9 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-23 04:15:49,492 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=44.1 M, priority=20, time=3840473817730621; duration=4sec 2015-08-23 04:16:45,649 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:16:46,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98772491, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/26289f873baa43cf8658e53325324d4d 2015-08-23 04:16:46,542 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/26289f873baa43cf8658e53325324d4d, entries=321331, sequenceid=98772491, filesize=8.6 M 2015-08-23 04:16:46,543 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221720, currentsize=112.85 KB/115560 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 894ms, sequenceid=98772491, compaction requested=true 2015-08-23 04:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21503033, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38579, evicted=173, evictedPerRun=0.0044843051582574844 2015-08-23 04:18:28,475 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.13:10110] 2015-08-23 04:18:28,534 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303227331 with entries=50112, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303508308 2015-08-23T04:21:21.049+0000: [CMS-concurrent-mark-start] 2015-08-23T04:21:21.136+0000: [CMS-concurrent-mark: 0.088/0.088 secs] [Times: user=0.46 sys=0.00, real=0.09 secs] 2015-08-23T04:21:21.137+0000: [CMS-concurrent-preclean-start] 2015-08-23T04:21:21.141+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T04:21:21.141+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T04:21:23.922+0000: [CMS-concurrent-abortable-preclean: 0.678/2.782 secs] [Times: user=0.89 sys=0.05, real=2.79 secs] 2015-08-23T04:21:23.957+0000: [CMS-concurrent-sweep-start] 2015-08-23T04:21:24.162+0000: [CMS-concurrent-sweep: 0.205/0.205 secs] [Times: user=0.23 sys=0.00, real=0.21 secs] 2015-08-23T04:21:24.162+0000: [CMS-concurrent-reset-start] 2015-08-23T04:21:24.183+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 04:22:13,326 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:22:14,156 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91569350, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/41804d6c79bd4472a50d9766478d3c1f 2015-08-23 04:22:14,217 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/41804d6c79bd4472a50d9766478d3c1f, entries=367254, sequenceid=91569350, filesize=8.7 M 2015-08-23 04:22:14,217 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222824, currentsize=115.77 KB/118552 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 891ms, sequenceid=91569350, compaction requested=true 2015-08-23 04:22:14,247 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 04:22:14,247 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=26.1 M 2015-08-23 04:22:14,257 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:22:17,373 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into c47d018b3ec54f4ea3c3339c4b7871f7(size=25.9 M), total size for store is 6.2 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-23 04:22:17,373 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=26.1 M, priority=20, time=3840862658180188; duration=3sec 2015-08-23 04:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21507077, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38609, evicted=173, evictedPerRun=0.004480820614844561 2015-08-23 04:23:07,304 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303508308 with entries=50049, filesize=121.73 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303787067 2015-08-23 04:23:40,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 3691 2015-08-23 04:23:44,567 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.23 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:23:45,007 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57618170, memsize=56.2 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/894a830bcac743ec86321a349398dc21 2015-08-23 04:23:45,059 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/894a830bcac743ec86321a349398dc21, entries=195431, sequenceid=57618170, filesize=3.6 M 2015-08-23 04:23:45,060 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~56.23 MB/58964992, currentsize=3.63 KB/3712 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 493ms, sequenceid=57618170, compaction requested=true 2015-08-23 04:23:45,062 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. 2015-08-23 04:23:45,062 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp, totalSize=87.0 M 2015-08-23 04:23:45,067 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:23:53,566 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into a87b0cbac36e4258a1e620c60ea26e6a(size=87.0 M), total size for store is 1.8 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-23 04:23:53,566 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., storeName=e, fileCount=3, fileSize=87.0 M, priority=20, time=3840953473823782; duration=8sec 2015-08-23 04:25:28,393 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:25:29,226 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102766086, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/54924e3b3ecb46c184ce69d6b8c5d511 2015-08-23 04:25:29,279 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/54924e3b3ecb46c184ce69d6b8c5d511, entries=365571, sequenceid=102766086, filesize=8.7 M 2015-08-23 04:25:29,280 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220904, currentsize=116.06 KB/118848 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 886ms, sequenceid=102766086, compaction requested=true 2015-08-23 04:25:29,282 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 04:25:29,282 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=43.2 M 2015-08-23 04:25:29,290 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:25:33,293 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 71ab675a203243c5a73b1a28c9e9ccb9(size=43.0 M), total size for store is 8.7 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-23 04:25:33,293 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=43.2 M, priority=19, time=3841057693273547; duration=4sec 2015-08-23 04:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21526514, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38639, evicted=173, evictedPerRun=0.0044773416593670845 2015-08-23 04:27:46,291 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303787067 with entries=51429, filesize=121.81 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304066076 2015-08-23 04:27:46,293 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300150127 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300150127 2015-08-23 04:27:46,391 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300431135 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300431135 2015-08-23 04:27:46,491 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300710269 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300710269 2015-08-23 04:27:46,591 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300988213 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440300988213 2015-08-23 04:27:46,691 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301267772 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301267772 2015-08-23 04:27:46,791 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301547805 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301547805 2015-08-23 04:27:46,891 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301828275 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440301828275 2015-08-23 04:27:46,991 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302109610 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302109610 2015-08-23 04:27:47,091 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302388331 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302388331 2015-08-23 04:27:47,191 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302669209 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302669209 2015-08-23 04:27:47,291 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302947455 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440302947455 2015-08-23 04:27:51,428 INFO [sync.2] wal.FSHLog: Slow sync cost: 219 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.10:10110] 2015-08-23 04:27:51,428 INFO [sync.4] wal.FSHLog: Slow sync cost: 199 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.10:10110] 2015-08-23 04:27:51,428 INFO [sync.3] wal.FSHLog: Slow sync cost: 201 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.10:10110] 2015-08-23 04:27:51,428 INFO [sync.0] wal.FSHLog: Slow sync cost: 163 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.10:10110] 2015-08-23 04:27:51,428 INFO [sync.1] wal.FSHLog: Slow sync cost: 153 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.10:10110] 2015-08-23 04:29:15,155 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:29:15,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98830534, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/64a9af1f59dc4d9e958d12e7b4510638 2015-08-23 04:29:15,962 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/64a9af1f59dc4d9e958d12e7b4510638, entries=322625, sequenceid=98830534, filesize=8.6 M 2015-08-23 04:29:15,962 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134235344, currentsize=122.27 KB/125208 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 807ms, sequenceid=98830534, compaction requested=true 2015-08-23 04:29:15,965 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 04:29:15,965 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=25.7 M 2015-08-23 04:29:15,974 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:29:18,584 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 809fe08c16f14c98b2c9ab94a7709662(size=25.4 M), total size for store is 7.4 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 04:29:18,584 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=25.7 M, priority=19, time=3841284376827543; duration=2sec 2015-08-23 04:29:41,307 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:29:42,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=100967592, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/3753bcc5636646458f8b874f51271f56 2015-08-23 04:29:42,214 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/3753bcc5636646458f8b874f51271f56, entries=384703, sequenceid=100967592, filesize=8.9 M 2015-08-23 04:29:42,215 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222328, currentsize=165.76 KB/169736 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 908ms, sequenceid=100967592, compaction requested=true 2015-08-23T04:31:08.183+0000: [CMS-concurrent-mark-start] 2015-08-23T04:31:08.229+0000: [CMS-concurrent-mark: 0.046/0.046 secs] [Times: user=0.24 sys=0.00, real=0.04 secs] 2015-08-23T04:31:08.229+0000: [CMS-concurrent-preclean-start] 2015-08-23T04:31:08.232+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T04:31:08.232+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T04:31:11.196+0000: [CMS-concurrent-abortable-preclean: 0.659/2.963 secs] [Times: user=0.89 sys=0.06, real=2.96 secs] 2015-08-23T04:31:11.231+0000: [CMS-concurrent-sweep-start] 2015-08-23T04:31:11.463+0000: [CMS-concurrent-sweep: 0.232/0.232 secs] [Times: user=0.27 sys=0.00, real=0.24 secs] 2015-08-23T04:31:11.463+0000: [CMS-concurrent-reset-start] 2015-08-23T04:31:11.484+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 04:32:27,714 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304066076 with entries=51827, filesize=121.74 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304347531 2015-08-23 04:32:27,716 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303227331 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303227331 2015-08-23 04:32:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21530807, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38669, evicted=173, evictedPerRun=0.004473867826163769 2015-08-23 04:37:07,127 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304347531 with entries=50080, filesize=121.76 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304626945 2015-08-23 04:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21530807, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38699, evicted=173, evictedPerRun=0.004470399580895901 2015-08-23 04:37:45,330 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:37:46,090 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91645866, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/0c26fb7fe14d4ae085408ddf76358003 2015-08-23 04:37:46,125 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/0c26fb7fe14d4ae085408ddf76358003, entries=367123, sequenceid=91645866, filesize=8.7 M 2015-08-23 04:37:46,126 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218320, currentsize=121.14 KB/124048 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 796ms, sequenceid=91645866, compaction requested=true 2015-08-23 04:38:22,216 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:38:22,977 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102835580, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/4dd24a4ee069478d9776ec764127bb9a 2015-08-23 04:38:23,012 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/4dd24a4ee069478d9776ec764127bb9a, entries=365540, sequenceid=102835580, filesize=8.7 M 2015-08-23 04:38:23,013 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223808, currentsize=76.69 KB/78528 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 797ms, sequenceid=102835580, compaction requested=true 2015-08-23 04:38:59,398 INFO [sync.0] wal.FSHLog: Slow sync cost: 114 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.2:10110] 2015-08-23 04:38:59,398 INFO [sync.4] wal.FSHLog: Slow sync cost: 146 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.2:10110] 2015-08-23 04:38:59,398 INFO [sync.1] wal.FSHLog: Slow sync cost: 109 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.2:10110] 2015-08-23 04:41:43,212 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:41:43,941 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98887434, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/185abdccbb30442cb331ca0144944879 2015-08-23 04:41:43,976 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/185abdccbb30442cb331ca0144944879, entries=321921, sequenceid=98887434, filesize=8.6 M 2015-08-23 04:41:43,977 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218200, currentsize=98.19 KB/100544 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 765ms, sequenceid=98887434, compaction requested=true 2015-08-23 04:41:46,033 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304626945 with entries=49159, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304905885 2015-08-23 04:41:46,036 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303508308 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303508308 2015-08-23 04:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21530807, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38729, evicted=173, evictedPerRun=0.00446693692356348 2015-08-23 04:43:37,208 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:43:37,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101040857, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/2d235157070440dd98412cd487babe0a 2015-08-23 04:43:37,955 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/2d235157070440dd98412cd487babe0a, entries=384952, sequenceid=101040857, filesize=8.9 M 2015-08-23 04:43:37,959 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219928, currentsize=127.55 KB/130616 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 751ms, sequenceid=101040857, compaction requested=true 2015-08-23 04:43:37,960 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 04:43:37,960 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=61.7 M 2015-08-23 04:43:37,968 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:43:43,246 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 8995449b15d94638ad7c3eb732ac9331(size=61.5 M), total size for store is 7.9 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 04:43:43,246 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=61.7 M, priority=20, time=3842146371738779; duration=5sec 2015-08-23T04:45:43.240+0000: [CMS-concurrent-mark-start] 2015-08-23T04:45:43.295+0000: [CMS-concurrent-mark: 0.056/0.056 secs] [Times: user=0.28 sys=0.00, real=0.06 secs] 2015-08-23T04:45:43.295+0000: [CMS-concurrent-preclean-start] 2015-08-23T04:45:43.299+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T04:45:43.299+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T04:45:46.139+0000: [CMS-concurrent-abortable-preclean: 0.537/2.840 secs] [Times: user=0.75 sys=0.05, real=2.84 secs] 2015-08-23T04:45:46.174+0000: [CMS-concurrent-sweep-start] 2015-08-23T04:45:46.397+0000: [CMS-concurrent-sweep: 0.223/0.223 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-23T04:45:46.397+0000: [CMS-concurrent-reset-start] 2015-08-23T04:45:46.419+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 04:46:25,837 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.4:10110] 2015-08-23 04:46:25,896 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304905885 with entries=48741, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305185654 2015-08-23 04:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21539958, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38759, evicted=173, evictedPerRun=0.00446347938850522 2015-08-23 04:51:04,418 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305185654 with entries=49694, filesize=121.81 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305464166 2015-08-23 04:51:16,136 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:51:16,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102903667, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/2de0ff28670748b4b2805e5cac522440 2015-08-23 04:51:17,005 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/2de0ff28670748b4b2805e5cac522440, entries=365385, sequenceid=102903667, filesize=8.7 M 2015-08-23 04:51:17,006 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221576, currentsize=85.85 KB/87912 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 870ms, sequenceid=102903667, compaction requested=true 2015-08-23 04:51:17,009 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 04:51:17,009 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=60.4 M 2015-08-23 04:51:17,018 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:51:22,502 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 894fe98f6f7f4ad49cac8bebf30c7573(size=60.2 M), total size for store is 8.7 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 04:51:22,502 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=60.4 M, priority=19, time=3842605420280061; duration=5sec 2015-08-23 04:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21549362, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38789, evicted=173, evictedPerRun=0.004460027441382408 2015-08-23 04:53:14,966 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:53:15,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91719355, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/7f80b59c5de049e9b18ec8fe208e1f0d 2015-08-23 04:53:15,784 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/7f80b59c5de049e9b18ec8fe208e1f0d, entries=366402, sequenceid=91719355, filesize=8.7 M 2015-08-23 04:53:15,785 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218104, currentsize=225.30 KB/230712 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 819ms, sequenceid=91719355, compaction requested=true 2015-08-23 04:53:15,787 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 04:53:15,787 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=43.3 M 2015-08-23 04:53:15,794 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:53:19,648 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into af10092d57cf4842a9d3d007d75313a8(size=43.1 M), total size for store is 6.3 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-23 04:53:19,648 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=43.3 M, priority=20, time=3842724198549045; duration=3sec 2015-08-23 04:54:14,055 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:54:14,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=98943319, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/d00a2743f8124e85bc51d84662028afb 2015-08-23 04:54:14,866 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/d00a2743f8124e85bc51d84662028afb, entries=322975, sequenceid=98943319, filesize=8.6 M 2015-08-23 04:54:14,867 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232192, currentsize=243.67 KB/249520 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 812ms, sequenceid=98943319, compaction requested=true 2015-08-23 04:54:14,869 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 04:54:14,869 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=42.6 M 2015-08-23 04:54:14,876 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 04:54:18,630 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 951bf087c5eb43f19f89246332d36e7e(size=42.3 M), total size for store is 7.4 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-23 04:54:18,630 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=42.6 M, priority=19, time=3842783280793776; duration=3sec 2015-08-23 04:55:44,056 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305464166 with entries=50145, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305743907 2015-08-23 04:57:34,072 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 04:57:34,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101113705, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/1ae5976ce8c4496d82524947b3685cbb 2015-08-23 04:57:34,920 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/1ae5976ce8c4496d82524947b3685cbb, entries=385111, sequenceid=101113705, filesize=8.9 M 2015-08-23 04:57:34,922 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223656, currentsize=117.52 KB/120344 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 850ms, sequenceid=101113705, compaction requested=true 2015-08-23 04:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21563197, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38819, evicted=173, evictedPerRun=0.004456580616533756 2015-08-23T04:59:00.060+0000: [CMS-concurrent-mark-start] 2015-08-23T04:59:00.117+0000: [CMS-concurrent-mark: 0.056/0.056 secs] [Times: user=0.29 sys=0.00, real=0.06 secs] 2015-08-23T04:59:00.117+0000: [CMS-concurrent-preclean-start] 2015-08-23T04:59:00.120+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T04:59:00.120+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T04:59:02.888+0000: [CMS-concurrent-abortable-preclean: 0.564/2.767 secs] [Times: user=0.79 sys=0.03, real=2.77 secs] 2015-08-23T04:59:02.921+0000: [CMS-concurrent-sweep-start] 2015-08-23T04:59:03.144+0000: [CMS-concurrent-sweep: 0.223/0.223 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T04:59:03.144+0000: [CMS-concurrent-reset-start] 2015-08-23T04:59:03.166+0000: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 05:00:25,420 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305743907 with entries=50168, filesize=121.75 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306025266 2015-08-23 05:01:37,457 INFO [sync.2] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.8:10110] 2015-08-23 05:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21563197, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38849, evicted=173, evictedPerRun=0.004453138913959265 2015-08-23 05:04:11,340 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:04:12,320 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=102972331, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/60079fe0a39a4c6eb6c2175e19882690 2015-08-23 05:04:12,356 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/60079fe0a39a4c6eb6c2175e19882690, entries=365300, sequenceid=102972331, filesize=8.7 M 2015-08-23 05:04:12,357 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222448, currentsize=168.16 KB/172200 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 1017ms, sequenceid=102972331, compaction requested=true 2015-08-23 05:05:04,467 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306025266 with entries=50612, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306304291 2015-08-23 05:05:09,642 INFO [sync.0] wal.FSHLog: Slow sync cost: 217 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:05:09,642 INFO [sync.1] wal.FSHLog: Slow sync cost: 197 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:05:09,642 INFO [sync.3] wal.FSHLog: Slow sync cost: 172 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:05:09,642 INFO [sync.2] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:05:09,642 INFO [sync.4] wal.FSHLog: Slow sync cost: 133 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:06:43,910 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:06:44,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99000705, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/9e4f2e220a54492cad0b709f401b4634 2015-08-23 05:06:44,673 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/9e4f2e220a54492cad0b709f401b4634, entries=321899, sequenceid=99000705, filesize=8.6 M 2015-08-23 05:06:44,674 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134226464, currentsize=203.67 KB/208560 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 764ms, sequenceid=99000705, compaction requested=true 2015-08-23 05:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21563197, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38879, evicted=173, evictedPerRun=0.004449702799320221 2015-08-23 05:08:47,630 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.05 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:08:48,390 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91794823, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/576ec0dbe8504316ab042d78434fc756 2015-08-23 05:08:48,427 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/576ec0dbe8504316ab042d78434fc756, entries=367544, sequenceid=91794823, filesize=8.7 M 2015-08-23 05:08:48,428 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134272688, currentsize=71.95 KB/73672 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 798ms, sequenceid=91794823, compaction requested=true 2015-08-23 05:09:04,955 INFO [sync.4] wal.FSHLog: Slow sync cost: 126 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:09:04,955 INFO [sync.3] wal.FSHLog: Slow sync cost: 144 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:09:16,059 INFO [sync.3] wal.FSHLog: Slow sync cost: 191 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.1:10110] 2015-08-23 05:09:44,130 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306304291 with entries=49865, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306583987 2015-08-23 05:11:35,428 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:11:36,258 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101187044, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/8fcdd27be6f74d6c988b2816fcd792e8 2015-08-23 05:11:36,303 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/8fcdd27be6f74d6c988b2816fcd792e8, entries=386278, sequenceid=101187044, filesize=8.9 M 2015-08-23 05:11:36,305 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134225456, currentsize=130.95 KB/134088 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 877ms, sequenceid=101187044, compaction requested=true 2015-08-23 05:11:36,307 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 05:11:36,307 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=79.3 M 2015-08-23 05:11:36,316 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:11:43,026 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into ed8fc0dbf5d84a60a5e0bdbae1345a41(size=79.1 M), total size for store is 7.9 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 05:11:43,026 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=79.3 M, priority=20, time=3843824718791003; duration=6sec 2015-08-23T05:12:22.822+0000: [CMS-concurrent-mark-start] 2015-08-23T05:12:22.879+0000: [CMS-concurrent-mark: 0.057/0.057 secs] [Times: user=0.30 sys=0.00, real=0.05 secs] 2015-08-23T05:12:22.879+0000: [CMS-concurrent-preclean-start] 2015-08-23T05:12:22.882+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T05:12:22.922+0000: [CMS-concurrent-sweep-start] 2015-08-23T05:12:23.149+0000: [CMS-concurrent-sweep: 0.227/0.227 secs] [Times: user=0.26 sys=0.00, real=0.22 secs] 2015-08-23T05:12:23.149+0000: [CMS-concurrent-reset-start] 2015-08-23T05:12:23.170+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 05:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21574953, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38909, evicted=173, evictedPerRun=0.004446272272616625 2015-08-23 05:14:27,010 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306583987 with entries=49805, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306866806 2015-08-23 05:17:07,532 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:17:08,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103040832, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/da86654007e544be82598c30f29f98e1 2015-08-23 05:17:08,462 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/da86654007e544be82598c30f29f98e1, entries=365279, sequenceid=103040832, filesize=8.7 M 2015-08-23 05:17:08,463 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219656, currentsize=175.60 KB/179816 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 931ms, sequenceid=103040832, compaction requested=true 2015-08-23 05:17:08,465 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 05:17:08,466 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=77.6 M 2015-08-23 05:17:08,473 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:17:15,227 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 65a9461ba3d04754b4682ba4efea6f90(size=77.4 M), total size for store is 8.7 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 05:17:15,227 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=77.6 M, priority=19, time=3844156876937095; duration=6sec 2015-08-23 05:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21587038, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38939, evicted=173, evictedPerRun=0.004442846402525902 2015-08-23 05:19:05,880 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.1:10110] 2015-08-23 05:19:05,939 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306866806 with entries=50140, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307145672 2015-08-23 05:19:13,730 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:19:14,514 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99057750, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/58faaccc3cc64c3fac52dcda9179cf4d 2015-08-23 05:19:14,575 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/58faaccc3cc64c3fac52dcda9179cf4d, entries=322765, sequenceid=99057750, filesize=8.6 M 2015-08-23 05:19:14,576 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218680, currentsize=102.72 KB/105184 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 846ms, sequenceid=99057750, compaction requested=true 2015-08-23 05:19:14,579 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 05:19:14,580 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=59.5 M 2015-08-23 05:19:14,587 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:19:19,790 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 935240502f3f4a73802ae4a28949bd16(size=59.2 M), total size for store is 7.4 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 05:19:19,790 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=59.5 M, priority=19, time=3844282991018927; duration=5sec 2015-08-23 05:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21597006, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38969, evicted=173, evictedPerRun=0.0044394261203706264 2015-08-23 05:23:47,385 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [172.24.32.16:10110, 172.24.32.1:10110, 172.24.32.12:10110] 2015-08-23 05:23:47,444 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307145672 with entries=50195, filesize=121.77 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307427132 2015-08-23 05:23:50,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 19910 2015-08-23 05:24:00,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 21586 2015-08-23 05:24:10,785 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.47 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:24:11,230 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57709940, memsize=56.5 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/e9325db379a14b818a79b75331c19f0d 2015-08-23 05:24:11,293 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/e9325db379a14b818a79b75331c19f0d, entries=196315, sequenceid=57709940, filesize=3.6 M 2015-08-23 05:24:11,294 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~56.47 MB/59209088, currentsize=13.79 KB/14120 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 509ms, sequenceid=57709940, compaction requested=true 2015-08-23 05:24:20,889 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:24:21,681 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91869226, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/2bbebd4f75ef46958f198470a14d1a39 2015-08-23 05:24:21,717 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/2bbebd4f75ef46958f198470a14d1a39, entries=367287, sequenceid=91869226, filesize=8.7 M 2015-08-23 05:24:21,718 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219656, currentsize=122.83 KB/125776 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 829ms, sequenceid=91869226, compaction requested=true 2015-08-23 05:24:21,720 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 05:24:21,720 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=60.5 M 2015-08-23 05:24:21,726 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:24:27,056 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 407ec7d39c994e04b77a2022fb96fb1e(size=60.3 M), total size for store is 6.3 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 05:24:27,056 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=60.5 M, priority=20, time=3844590131261702; duration=5sec 2015-08-23T05:25:37.286+0000: [CMS-concurrent-mark-start] 2015-08-23T05:25:37.357+0000: [CMS-concurrent-mark: 0.071/0.071 secs] [Times: user=0.37 sys=0.00, real=0.07 secs] 2015-08-23T05:25:37.357+0000: [CMS-concurrent-preclean-start] 2015-08-23T05:25:37.361+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T05:25:37.361+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23 05:25:39,213 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23T05:25:39.485+0000: [CMS-concurrent-abortable-preclean: 0.487/2.125 secs] [Times: user=0.88 sys=0.04, real=2.13 secs] 2015-08-23T05:25:39.516+0000: [CMS-concurrent-sweep-start] 2015-08-23T05:25:39.729+0000: [CMS-concurrent-sweep: 0.213/0.213 secs] [Times: user=0.46 sys=0.00, real=0.21 secs] 2015-08-23T05:25:39.729+0000: [CMS-concurrent-reset-start] 2015-08-23T05:25:39.750+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 2015-08-23 05:25:40,064 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101260448, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/149265ae58714611904de64c018be186 2015-08-23 05:25:40,108 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/149265ae58714611904de64c018be186, entries=386229, sequenceid=101260448, filesize=8.9 M 2015-08-23 05:25:40,110 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221408, currentsize=164.21 KB/168152 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 897ms, sequenceid=101260448, compaction requested=true 2015-08-23 05:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21606390, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=38999, evicted=173, evictedPerRun=0.0044360109604895115 2015-08-23 05:28:28,383 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307427132 with entries=49761, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307708245 2015-08-23 05:28:28,385 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303787067 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440303787067 2015-08-23 05:28:28,442 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304066076 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304066076 2015-08-23 05:28:28,491 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304347531 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304347531 2015-08-23 05:28:28,541 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304626945 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304626945 2015-08-23 05:28:28,591 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304905885 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440304905885 2015-08-23 05:28:28,641 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305185654 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305185654 2015-08-23 05:28:28,691 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305464166 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305464166 2015-08-23 05:28:28,741 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305743907 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440305743907 2015-08-23 05:28:28,791 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306025266 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306025266 2015-08-23 05:28:28,841 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306304291 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306304291 2015-08-23 05:28:28,891 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306583987 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306583987 2015-08-23 05:30:03,453 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:30:04,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103109564, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/b01caac8a88b42a5956b9fee192d522f 2015-08-23 05:30:04,272 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/b01caac8a88b42a5956b9fee192d522f, entries=364962, sequenceid=103109564, filesize=8.7 M 2015-08-23 05:30:04,274 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219432, currentsize=127.45 KB/130512 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 821ms, sequenceid=103109564, compaction requested=true 2015-08-23 05:30:58,380 INFO [sync.2] wal.FSHLog: Slow sync cost: 174 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.15:10110] 2015-08-23 05:31:19,916 INFO [sync.0] wal.FSHLog: Slow sync cost: 111 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.15:10110] 2015-08-23 05:31:19,916 INFO [sync.4] wal.FSHLog: Slow sync cost: 126 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.15:10110] 2015-08-23 05:31:43,655 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:31:44,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99114204, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/1427e54393a644dcb60b485d70fd1e4c 2015-08-23 05:31:44,415 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/1427e54393a644dcb60b485d70fd1e4c, entries=321974, sequenceid=99114204, filesize=8.6 M 2015-08-23 05:31:44,416 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134233432, currentsize=181.07 KB/185416 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 761ms, sequenceid=99114204, compaction requested=true 2015-08-23 05:32:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21606390, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39029, evicted=173, evictedPerRun=0.004432601388543844 2015-08-23 05:33:08,004 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307708245 with entries=48856, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307987914 2015-08-23 05:33:08,006 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306866806 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440306866806 2015-08-23 05:33:08,037 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307145672 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307145672 2015-08-23T05:37:33.402+0000: [CMS-concurrent-mark-start] 2015-08-23T05:37:33.490+0000: [CMS-concurrent-mark: 0.088/0.088 secs] [Times: user=0.45 sys=0.00, real=0.08 secs] 2015-08-23T05:37:33.490+0000: [CMS-concurrent-preclean-start] 2015-08-23T05:37:33.494+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T05:37:33.495+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T05:37:36.415+0000: [CMS-concurrent-abortable-preclean: 1.617/2.920 secs] [Times: user=1.88 sys=0.01, real=2.92 secs] 2015-08-23T05:37:36.454+0000: [CMS-concurrent-sweep-start] 2015-08-23T05:37:36.659+0000: [CMS-concurrent-sweep: 0.205/0.205 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] 2015-08-23T05:37:36.659+0000: [CMS-concurrent-reset-start] 2015-08-23T05:37:36.680+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 05:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21606390, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39059, evicted=173, evictedPerRun=0.004429196938872337 2015-08-23 05:37:47,166 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307987914 with entries=49144, filesize=121.70 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308266790 2015-08-23 05:39:41,229 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:39:42,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101332919, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/d6feaed0198744fdb875e21e50ac1543 2015-08-23 05:39:42,113 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/d6feaed0198744fdb875e21e50ac1543, entries=386292, sequenceid=101332919, filesize=8.9 M 2015-08-23 05:39:42,114 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222224, currentsize=138.23 KB/141552 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 885ms, sequenceid=101332919, compaction requested=true 2015-08-23 05:39:42,116 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 05:39:42,116 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=96.9 M 2015-08-23 05:39:42,125 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:39:50,345 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into c0a40190aa854053923d13f48de8d072(size=96.7 M), total size for store is 8.0 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-23 05:39:50,345 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=96.9 M, priority=20, time=3845510527839719; duration=8sec 2015-08-23 05:39:53,452 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:39:54,228 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=91942938, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/7c5c4b4dba55487c86f42adf511660ca 2015-08-23 05:39:54,264 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/7c5c4b4dba55487c86f42adf511660ca, entries=367404, sequenceid=91942938, filesize=8.7 M 2015-08-23 05:39:54,265 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223248, currentsize=96.48 KB/98800 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 813ms, sequenceid=91942938, compaction requested=true 2015-08-23 05:42:27,804 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308266790 with entries=50184, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308547648 2015-08-23 05:42:32,958 INFO [sync.4] wal.FSHLog: Slow sync cost: 189 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.12:10110] 2015-08-23 05:42:32,958 INFO [sync.2] wal.FSHLog: Slow sync cost: 212 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.12:10110] 2015-08-23 05:42:32,958 INFO [sync.1] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.12:10110] 2015-08-23 05:42:32,958 INFO [sync.3] wal.FSHLog: Slow sync cost: 192 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.12:10110] 2015-08-23 05:42:32,958 INFO [sync.0] wal.FSHLog: Slow sync cost: 148 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.12:10110] 2015-08-23 05:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21620748, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39089, evicted=173, evictedPerRun=0.004425797611474991 2015-08-23 05:42:59,656 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:43:00,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103177221, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/31313f61f6184d9b8ca5b1ecd74e6c0d 2015-08-23 05:43:00,484 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/31313f61f6184d9b8ca5b1ecd74e6c0d, entries=365035, sequenceid=103177221, filesize=8.7 M 2015-08-23 05:43:00,485 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134226768, currentsize=123.07 KB/126024 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 829ms, sequenceid=103177221, compaction requested=true 2015-08-23 05:43:00,487 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 05:43:00,487 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 4 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=206.6 M 2015-08-23 05:43:00,495 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:43:17,692 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 4 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 6ff404e527bb413eaf0cf6fabee09838(size=206.6 M), total size for store is 8.7 G. This selection was in queue for 0sec, and took 17sec to execute. 2015-08-23 05:43:17,692 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=4, fileSize=206.6 M, priority=19, time=3845708898806306; duration=17sec 2015-08-23 05:44:09,866 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:44:10,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99170675, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/1ceec756c71f48b9bfd1917ff4566102 2015-08-23 05:44:10,633 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/1ceec756c71f48b9bfd1917ff4566102, entries=322062, sequenceid=99170675, filesize=8.6 M 2015-08-23 05:44:10,634 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217840, currentsize=119.85 KB/122728 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 769ms, sequenceid=99170675, compaction requested=true 2015-08-23 05:44:10,636 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 05:44:10,636 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=76.4 M 2015-08-23 05:44:10,643 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:44:17,165 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into e48746de1cdf45b6b9efd132b8f22fb8(size=76.1 M), total size for store is 7.4 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 05:44:17,165 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=76.4 M, priority=19, time=3845779047933450; duration=6sec 2015-08-23 05:47:08,357 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308547648 with entries=50927, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308828275 2015-08-23T05:47:18.200+0000: [CMS-concurrent-mark-start] 2015-08-23T05:47:18.260+0000: [CMS-concurrent-mark: 0.059/0.059 secs] [Times: user=0.31 sys=0.00, real=0.06 secs] 2015-08-23T05:47:18.260+0000: [CMS-concurrent-preclean-start] 2015-08-23T05:47:18.263+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T05:47:18.295+0000: [CMS-concurrent-sweep-start] 2015-08-23T05:47:18.516+0000: [CMS-concurrent-sweep: 0.219/0.221 secs] [Times: user=0.24 sys=0.00, real=0.22 secs] 2015-08-23T05:47:18.516+0000: [CMS-concurrent-reset-start] 2015-08-23T05:47:18.538+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 05:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21665744, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39119, evicted=173, evictedPerRun=0.004422403406351805 2015-08-23 05:51:46,435 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308828275 with entries=50431, filesize=121.65 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309106233 2015-08-23 05:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21665744, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39149, evicted=173, evictedPerRun=0.004419014323502779 2015-08-23 05:53:43,151 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:53:44,171 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101407295, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/9141d9d5bc0a43fbbcd7f8d3f4725c76 2015-08-23 05:53:44,216 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/9141d9d5bc0a43fbbcd7f8d3f4725c76, entries=385894, sequenceid=101407295, filesize=8.9 M 2015-08-23 05:53:44,217 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219816, currentsize=187.77 KB/192272 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 1066ms, sequenceid=101407295, compaction requested=true 2015-08-23 05:55:23,707 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:55:24,473 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=92019255, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/bd514ab8cf894bde9a31188ce313941f 2015-08-23 05:55:24,517 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/bd514ab8cf894bde9a31188ce313941f, entries=367791, sequenceid=92019255, filesize=8.7 M 2015-08-23 05:55:24,518 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218928, currentsize=156.29 KB/160040 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 811ms, sequenceid=92019255, compaction requested=true 2015-08-23 05:55:24,520 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 05:55:24,520 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=77.7 M 2015-08-23 05:55:24,527 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 05:55:31,232 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into aebbaceeaa534488af441e0989f3c89b(size=77.6 M), total size for store is 6.3 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-08-23 05:55:31,232 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=77.7 M, priority=20, time=3846452931704081; duration=6sec 2015-08-23 05:55:56,314 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:55:57,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103246617, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/cc725f73f1784fa395d89ea5a31ee572 2015-08-23 05:55:57,137 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/cc725f73f1784fa395d89ea5a31ee572, entries=364950, sequenceid=103246617, filesize=8.7 M 2015-08-23 05:55:57,138 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222520, currentsize=100.91 KB/103336 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 824ms, sequenceid=103246617, compaction requested=true 2015-08-23 05:56:27,663 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309106233 with entries=51341, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309387514 2015-08-23 05:56:40,877 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.07 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 05:56:41,598 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99229071, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/c3ee9174061c4124a7e51083c1f140ec 2015-08-23 05:56:41,659 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/c3ee9174061c4124a7e51083c1f140ec, entries=322489, sequenceid=99229071, filesize=8.6 M 2015-08-23 05:56:41,660 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134294568, currentsize=170.70 KB/174800 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 783ms, sequenceid=99229071, compaction requested=true 2015-08-23 05:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21677802, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39179, evicted=173, evictedPerRun=0.004415630828589201 2015-08-23T06:00:31.221+0000: [CMS-concurrent-mark-start] 2015-08-23T06:00:31.279+0000: [CMS-concurrent-mark: 0.058/0.058 secs] [Times: user=0.31 sys=0.00, real=0.06 secs] 2015-08-23T06:00:31.279+0000: [CMS-concurrent-preclean-start] 2015-08-23T06:00:31.282+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T06:00:31.282+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T06:00:34.318+0000: [CMS-concurrent-abortable-preclean: 0.932/3.035 secs] [Times: user=1.18 sys=0.04, real=3.03 secs] 2015-08-23T06:00:34.350+0000: [CMS-concurrent-sweep-start] 2015-08-23T06:00:34.573+0000: [CMS-concurrent-sweep: 0.223/0.223 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T06:00:34.573+0000: [CMS-concurrent-reset-start] 2015-08-23T06:00:34.594+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 06:01:07,669 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309387514 with entries=50363, filesize=121.66 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309667451 2015-08-23 06:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21677802, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39209, evicted=173, evictedPerRun=0.004412252455949783 2015-08-23 06:02:50,490 INFO [sync.1] wal.FSHLog: Slow sync cost: 184 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.8:10110] 2015-08-23 06:02:50,490 INFO [sync.0] wal.FSHLog: Slow sync cost: 205 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.8:10110] 2015-08-23 06:02:50,490 INFO [sync.2] wal.FSHLog: Slow sync cost: 141 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.8:10110] 2015-08-23 06:02:50,490 INFO [sync.3] wal.FSHLog: Slow sync cost: 125 ms, current pipeline: [172.24.32.16:10110, 172.24.32.12:10110, 172.24.32.8:10110] 2015-08-23 06:05:46,982 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309667451 with entries=49436, filesize=121.62 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309946814 2015-08-23 06:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21677802, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39239, evicted=173, evictedPerRun=0.004408878739923239 2015-08-23 06:07:44,031 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:07:44,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101480720, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/300ffebe062946f2b47932f4c5f4f6ff 2015-08-23 06:07:44,897 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/300ffebe062946f2b47932f4c5f4f6ff, entries=385885, sequenceid=101480720, filesize=8.9 M 2015-08-23 06:07:44,899 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220464, currentsize=123.66 KB/126632 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 868ms, sequenceid=101480720, compaction requested=true 2015-08-23 06:07:44,901 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 06:07:44,901 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=114.5 M 2015-08-23 06:07:44,911 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:07:54,429 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into f9c979a917d645e2bf8c002b9a8c9ef2(size=114.4 M), total size for store is 8.0 G. This selection was in queue for 0sec, and took 9sec to execute. 2015-08-23 06:07:54,429 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=114.5 M, priority=20, time=3847193312702219; duration=9sec 2015-08-23 06:08:51,853 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:08:52,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103315234, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/bd2128bf89e143be8ed32fb6249df352 2015-08-23 06:08:52,697 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/bd2128bf89e143be8ed32fb6249df352, entries=365275, sequenceid=103315234, filesize=8.7 M 2015-08-23 06:08:52,698 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134222800, currentsize=106.70 KB/109256 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 845ms, sequenceid=103315234, compaction requested=true 2015-08-23 06:09:08,506 INFO [sync.2] wal.FSHLog: Slow sync cost: 197 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.8:10110] 2015-08-23 06:09:08,506 INFO [sync.1] wal.FSHLog: Slow sync cost: 200 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.8:10110] 2015-08-23 06:09:08,506 INFO [sync.0] wal.FSHLog: Slow sync cost: 238 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.8:10110] 2015-08-23 06:09:08,506 INFO [sync.3] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.8:10110] 2015-08-23 06:09:08,506 INFO [sync.4] wal.FSHLog: Slow sync cost: 141 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.8:10110] 2015-08-23 06:09:08,976 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:09:09,695 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99285624, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/657a506f00bb4dd882dab2fadddff60b 2015-08-23 06:09:09,731 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/657a506f00bb4dd882dab2fadddff60b, entries=322363, sequenceid=99285624, filesize=8.5 M 2015-08-23 06:09:09,732 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134226320, currentsize=102.59 KB/105048 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 756ms, sequenceid=99285624, compaction requested=true 2015-08-23 06:09:09,735 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 06:09:09,735 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=93.2 M 2015-08-23 06:09:09,741 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:09:17,246 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into 87f1efe6a023406991082976504310eb(size=93.0 M), total size for store is 7.4 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-08-23 06:09:17,246 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=93.2 M, priority=19, time=3847278146591288; duration=7sec 2015-08-23 06:10:27,255 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309946814 with entries=49968, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310227107 2015-08-23 06:10:32,423 INFO [sync.1] wal.FSHLog: Slow sync cost: 173 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.7:10110] 2015-08-23 06:10:32,423 INFO [sync.0] wal.FSHLog: Slow sync cost: 178 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.7:10110] 2015-08-23 06:10:32,424 INFO [sync.3] wal.FSHLog: Slow sync cost: 218 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.7:10110] 2015-08-23 06:10:32,424 INFO [sync.4] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.7:10110] 2015-08-23 06:10:32,424 INFO [sync.2] wal.FSHLog: Slow sync cost: 118 ms, current pipeline: [172.24.32.16:10110, 172.24.32.14:10110, 172.24.32.7:10110] 2015-08-23 06:10:53,229 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:10:54,023 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=92093411, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/d6466b765456418b8a83d0a57d0d0642 2015-08-23 06:10:54,059 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/d6466b765456418b8a83d0a57d0d0642, entries=367621, sequenceid=92093411, filesize=8.7 M 2015-08-23 06:10:54,060 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134239208, currentsize=128.78 KB/131872 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 831ms, sequenceid=92093411, compaction requested=true 2015-08-23 06:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21710423, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39269, evicted=173, evictedPerRun=0.004405510611832142 2015-08-23T06:13:41.041+0000: [CMS-concurrent-mark-start] 2015-08-23T06:13:41.097+0000: [CMS-concurrent-mark: 0.056/0.056 secs] [Times: user=0.29 sys=0.00, real=0.05 secs] 2015-08-23T06:13:41.097+0000: [CMS-concurrent-preclean-start] 2015-08-23T06:13:41.101+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T06:13:41.130+0000: [CMS-concurrent-sweep-start] 2015-08-23T06:13:41.356+0000: [CMS-concurrent-sweep: 0.227/0.227 secs] [Times: user=0.25 sys=0.00, real=0.23 secs] 2015-08-23T06:13:41.357+0000: [CMS-concurrent-reset-start] 2015-08-23T06:13:41.377+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 06:15:06,326 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310227107 with entries=49152, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310506150 2015-08-23 06:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21710423, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39299, evicted=173, evictedPerRun=0.004402147606015205 2015-08-23 06:18:40,963 INFO [sync.2] wal.FSHLog: Slow sync cost: 154 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-23 06:18:40,963 INFO [sync.1] wal.FSHLog: Slow sync cost: 171 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-23 06:18:40,963 INFO [sync.3] wal.FSHLog: Slow sync cost: 133 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-23 06:18:40,963 INFO [sync.4] wal.FSHLog: Slow sync cost: 118 ms, current pipeline: [172.24.32.16:10110, 172.24.32.15:10110, 172.24.32.8:10110] 2015-08-23 06:19:46,263 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310506150 with entries=49449, filesize=121.68 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310786090 2015-08-23 06:21:41,234 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:21:41,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99341456, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/593b36c297d64b6a900798e053824e7f 2015-08-23 06:21:41,985 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/593b36c297d64b6a900798e053824e7f, entries=323011, sequenceid=99341456, filesize=8.6 M 2015-08-23 06:21:41,986 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134225888, currentsize=82.16 KB/84128 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 752ms, sequenceid=99341456, compaction requested=true 2015-08-23 06:21:43,748 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:21:44,617 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101553432, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/7158c85cc46c4188a224d9431da967f8 2015-08-23 06:21:44,675 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/7158c85cc46c4188a224d9431da967f8, entries=385741, sequenceid=101553432, filesize=8.9 M 2015-08-23 06:21:44,676 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221032, currentsize=116.51 KB/119304 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 928ms, sequenceid=101553432, compaction requested=true 2015-08-23 06:21:49,420 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:21:50,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103383561, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/0bd2462307de4b6ab3f32f0523437fd3 2015-08-23 06:21:50,285 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/0bd2462307de4b6ab3f32f0523437fd3, entries=365400, sequenceid=103383561, filesize=8.7 M 2015-08-23 06:21:50,286 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221048, currentsize=122.99 KB/125944 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 866ms, sequenceid=103383561, compaction requested=true 2015-08-23 06:21:50,288 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 06:21:50,289 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=26.1 M 2015-08-23 06:21:50,298 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:21:52,808 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into a085ba34df6c4f6c820eacf1fef4adf9(size=25.8 M), total size for store is 8.8 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 06:21:52,808 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=26.1 M, priority=19, time=3848038699984805; duration=2sec 2015-08-23 06:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21714478, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39329, evicted=173, evictedPerRun=0.004398789722472429 2015-08-23 06:24:20,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 19150 2015-08-23 06:24:28,674 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310786090 with entries=49425, filesize=121.63 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311068506 2015-08-23 06:24:30,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 18898 2015-08-23 06:24:40,025 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 56.69 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:24:40,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=57801953, memsize=56.7 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp/7ec01287cb0642c18b4d6d4bff8efc43 2015-08-23 06:24:40,504 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/e/7ec01287cb0642c18b4d6d4bff8efc43, entries=196692, sequenceid=57801953, filesize=3.6 M 2015-08-23 06:24:40,505 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~56.69 MB/59441624, currentsize=4.22 KB/4320 for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. in 480ms, sequenceid=57801953, compaction requested=true 2015-08-23 06:24:40,507 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. 2015-08-23 06:24:40,507 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/a1233a97ab54c0f50ac9a39cd05f75a6/.tmp, totalSize=94.2 M 2015-08-23 06:24:40,511 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:24:43,426 INFO [sync.0] wal.FSHLog: Slow sync cost: 178 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:24:43,427 INFO [sync.2] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:24:43,427 INFO [sync.1] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:24:49,284 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. into a4cd10ae903c45a1b3530bbc3cd2c26a(size=94.2 M), total size for store is 1.8 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-23 06:24:49,284 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., storeName=e, fileCount=3, fileSize=94.2 M, priority=20, time=3848208918434720; duration=8sec 2015-08-23 06:26:23,551 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.04 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:26:24,352 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=92167204, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/0978085f2434478c96fdc7f228606583 2015-08-23 06:26:24,397 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/0978085f2434478c96fdc7f228606583, entries=367330, sequenceid=92167204, filesize=8.7 M 2015-08-23 06:26:24,398 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.04 MB/134261952, currentsize=87.64 KB/89744 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 847ms, sequenceid=92167204, compaction requested=true 2015-08-23 06:26:24,400 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 06:26:24,401 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 4 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=207.3 M 2015-08-23 06:26:24,410 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:26:41,470 INFO [sync.1] wal.FSHLog: Slow sync cost: 104 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:26:41,496 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 4 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into cd229e40fc344fa583451501862e16db(size=207.4 M), total size for store is 6.3 G. This selection was in queue for 0sec, and took 17sec to execute. 2015-08-23 06:26:41,497 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=4, fileSize=207.3 M, priority=20, time=3848312811981327; duration=17sec 2015-08-23T06:26:55.380+0000: [CMS-concurrent-mark-start] 2015-08-23T06:26:55.428+0000: [CMS-concurrent-mark: 0.048/0.048 secs] [Times: user=0.25 sys=0.00, real=0.05 secs] 2015-08-23T06:26:55.428+0000: [CMS-concurrent-preclean-start] 2015-08-23T06:26:55.432+0000: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2015-08-23T06:26:55.461+0000: [CMS-concurrent-sweep-start] 2015-08-23T06:26:55.694+0000: [CMS-concurrent-sweep: 0.232/0.232 secs] [Times: user=0.27 sys=0.00, real=0.24 secs] 2015-08-23T06:26:55.694+0000: [CMS-concurrent-reset-start] 2015-08-23T06:26:55.715+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 06:27:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21760356, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39359, evicted=173, evictedPerRun=0.0043954369612038136 2015-08-23 06:28:40,272 INFO [sync.2] wal.FSHLog: Slow sync cost: 124 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:28:40,272 INFO [sync.4] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms, current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110] 2015-08-23 06:29:07,470 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311068506 with entries=49146, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311347269 2015-08-23 06:29:07,470 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307427132 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307427132 2015-08-23 06:29:07,536 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307708245 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307708245 2015-08-23 06:29:07,603 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307987914 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440307987914 2015-08-23 06:29:07,669 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308266790 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308266790 2015-08-23 06:29:07,736 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308547648 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308547648 2015-08-23 06:29:07,803 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308828275 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440308828275 2015-08-23 06:29:07,869 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309106233 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309106233 2015-08-23 06:29:07,936 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309387514 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309387514 2015-08-23 06:29:08,003 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309667451 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309667451 2015-08-23 06:29:08,069 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309946814 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440309946814 2015-08-23 06:29:08,136 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310227107 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310227107 2015-08-23 06:29:08,203 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310506150 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310506150 2015-08-23 06:32:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21760356, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39389, evicted=173, evictedPerRun=0.004392089322209358 2015-08-23 06:33:47,330 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311347269 with entries=49115, filesize=121.68 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311627216 2015-08-23 06:33:52,546 INFO [sync.3] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.14:10110] 2015-08-23 06:33:52,546 INFO [sync.2] wal.FSHLog: Slow sync cost: 201 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.14:10110] 2015-08-23 06:33:52,546 INFO [sync.1] wal.FSHLog: Slow sync cost: 216 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.14:10110] 2015-08-23 06:33:52,546 INFO [sync.4] wal.FSHLog: Slow sync cost: 137 ms, current pipeline: [172.24.32.16:10110, 172.24.32.11:10110, 172.24.32.14:10110] 2015-08-23 06:34:09,115 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:34:09,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99397129, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/dae077aed89d49f7af02ebee28ed1855 2015-08-23 06:34:09,884 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/dae077aed89d49f7af02ebee28ed1855, entries=321881, sequenceid=99397129, filesize=8.6 M 2015-08-23 06:34:09,885 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221496, currentsize=106.20 KB/108752 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 770ms, sequenceid=99397129, compaction requested=true 2015-08-23 06:34:09,887 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 06:34:09,887 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=110.2 M 2015-08-23 06:34:09,894 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:34:18,699 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into fcdf3b54d56e42699452854c456067cb(size=109.9 M), total size for store is 7.5 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-08-23 06:34:18,700 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=110.2 M, priority=19, time=3848778298603637; duration=8sec 2015-08-23 06:34:52,130 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:34:52,878 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103451711, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/06795ba5127f42a38c02abde97629df2 2015-08-23 06:34:52,897 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/06795ba5127f42a38c02abde97629df2, entries=366647, sequenceid=103451711, filesize=8.7 M 2015-08-23 06:34:52,898 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134223584, currentsize=70.08 KB/71760 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 768ms, sequenceid=103451711, compaction requested=true 2015-08-23 06:35:45,070 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:35:45,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101626113, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/9cc1a7bd29ad4f81856a01d8823ad3a5 2015-08-23 06:35:45,952 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/9cc1a7bd29ad4f81856a01d8823ad3a5, entries=385829, sequenceid=101626113, filesize=8.9 M 2015-08-23 06:35:45,953 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218848, currentsize=176.96 KB/181208 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 883ms, sequenceid=101626113, compaction requested=true 2015-08-23 06:37:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21778856, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39419, evicted=173, evictedPerRun=0.004388746339827776 2015-08-23 06:38:29,609 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311627216 with entries=50421, filesize=121.65 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311909434 2015-08-23 06:38:29,611 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Archiving hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310786090 to hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/oldWALs/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440310786090 2015-08-23 06:41:53,187 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23T06:41:53.918+0000: [CMS-concurrent-mark-start] 2015-08-23T06:41:53.998+0000: [CMS-concurrent-mark: 0.080/0.080 secs] [Times: user=0.45 sys=0.00, real=0.08 secs] 2015-08-23T06:41:53.998+0000: [CMS-concurrent-preclean-start] 2015-08-23T06:41:54.001+0000: [CMS-concurrent-preclean: 0.003/0.004 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T06:41:54.001+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23 06:41:54,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=92240982, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/1cef11451c4345c8a7e9b62421a34c38 2015-08-23 06:41:54,057 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/1cef11451c4345c8a7e9b62421a34c38, entries=367091, sequenceid=92240982, filesize=8.7 M 2015-08-23 06:41:54,058 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218768, currentsize=138.80 KB/142136 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 871ms, sequenceid=92240982, compaction requested=true 2015-08-23T06:41:56.376+0000: [CMS-concurrent-abortable-preclean: 0.272/2.374 secs] [Times: user=0.47 sys=0.05, real=2.37 secs] 2015-08-23T06:41:56.419+0000: [CMS-concurrent-sweep-start] 2015-08-23T06:41:56.624+0000: [CMS-concurrent-sweep: 0.204/0.204 secs] [Times: user=0.22 sys=0.00, real=0.21 secs] 2015-08-23T06:41:56.624+0000: [CMS-concurrent-reset-start] 2015-08-23T06:41:56.644+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 06:42:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21778856, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39449, evicted=173, evictedPerRun=0.004385408945381641 2015-08-23 06:43:09,405 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440311909434 with entries=50382, filesize=121.80 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440312189128 2015-08-23 06:46:43,173 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.05 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:46:43,977 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99455705, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/a6cc6320a02349c785b110403f42d3c1 2015-08-23 06:46:44,029 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/a6cc6320a02349c785b110403f42d3c1, entries=323524, sequenceid=99455705, filesize=8.6 M 2015-08-23 06:46:44,030 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134267528, currentsize=225.85 KB/231272 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 857ms, sequenceid=99455705, compaction requested=true 2015-08-23 06:47:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21778856, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39479, evicted=173, evictedPerRun=0.004382076673209667 2015-08-23 06:47:47,970 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:47:48,775 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103521334, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/773de76cbabe4d868bf64eb32c691581 2015-08-23 06:47:48,828 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/773de76cbabe4d868bf64eb32c691581, entries=365272, sequenceid=103521334, filesize=8.7 M 2015-08-23 06:47:48,830 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134220504, currentsize=97.84 KB/100184 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 860ms, sequenceid=103521334, compaction requested=true 2015-08-23 06:47:48,832 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 06:47:48,832 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=43.2 M 2015-08-23 06:47:48,839 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:47:49,468 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440312189128 with entries=51082, filesize=121.71 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440312469267 2015-08-23 06:47:52,851 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into f6e2305209c84a2886df5b1a8887605f(size=43.0 M), total size for store is 8.8 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-08-23 06:47:52,852 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=43.2 M, priority=19, time=3849597243465564; duration=4sec 2015-08-23 06:49:45,209 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:49:46,046 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101699810, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/7b167c34556a4750b56fb28075f3bd5a 2015-08-23 06:49:46,099 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/7b167c34556a4750b56fb28075f3bd5a, entries=386394, sequenceid=101699810, filesize=8.9 M 2015-08-23 06:49:46,100 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134225952, currentsize=163.02 KB/166928 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 891ms, sequenceid=101699810, compaction requested=true 2015-08-23 06:49:46,102 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 06:49:46,102 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=26.6 M 2015-08-23 06:49:46,113 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:49:48,855 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into 7c88675e52f441c7a470c292914772f5(size=26.3 M), total size for store is 8.0 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 06:49:48,855 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=26.6 M, priority=19, time=3849714513846647; duration=2sec 2015-08-23T06:52:07.904+0000: [CMS-concurrent-mark-start] 2015-08-23T06:52:07.962+0000: [CMS-concurrent-mark: 0.058/0.058 secs] [Times: user=0.29 sys=0.00, real=0.06 secs] 2015-08-23T06:52:07.962+0000: [CMS-concurrent-preclean-start] 2015-08-23T06:52:07.966+0000: [CMS-concurrent-preclean: 0.003/0.004 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2015-08-23T06:52:07.995+0000: [CMS-concurrent-sweep-start] 2015-08-23T06:52:08.215+0000: [CMS-concurrent-sweep: 0.220/0.220 secs] [Times: user=0.25 sys=0.00, real=0.22 secs] 2015-08-23T06:52:08.215+0000: [CMS-concurrent-reset-start] 2015-08-23T06:52:08.239+0000: [CMS-concurrent-reset: 0.022/0.023 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2015-08-23 06:52:30,896 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 107 ms, current pipeline: [172.24.32.16:10110, 172.24.32.5:10110, 172.24.32.7:10110] 2015-08-23 06:52:30,955 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440312469267 with entries=50418, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440312750713 2015-08-23 06:52:36,073 INFO [sync.0] wal.FSHLog: Slow sync cost: 166 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.7:10110] 2015-08-23 06:52:36,073 INFO [sync.4] wal.FSHLog: Slow sync cost: 203 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.7:10110] 2015-08-23 06:52:36,073 INFO [sync.3] wal.FSHLog: Slow sync cost: 206 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.7:10110] 2015-08-23 06:52:36,074 INFO [sync.2] wal.FSHLog: Slow sync cost: 149 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.7:10110] 2015-08-23 06:52:36,074 INFO [sync.1] wal.FSHLog: Slow sync cost: 165 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.7:10110] 2015-08-23 06:52:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21789520, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39509, evicted=173, evictedPerRun=0.004378749057650566 2015-08-23 06:57:11,667 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440312750713 with entries=50031, filesize=121.71 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313031390 2015-08-23 06:57:22,930 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:57:23,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=92316001, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/6267aa271ecd4dd3a561a76b0f7c2235 2015-08-23 06:57:23,796 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/6267aa271ecd4dd3a561a76b0f7c2235, entries=367651, sequenceid=92316001, filesize=8.7 M 2015-08-23 06:57:23,797 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134240048, currentsize=147.30 KB/150840 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 867ms, sequenceid=92316001, compaction requested=true 2015-08-23 06:57:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21789520, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39539, evicted=173, evictedPerRun=0.004375426564365625 2015-08-23 06:59:11,250 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 06:59:11,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99512270, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/3e89c9e369cf4b3a92f291c8446086c0 2015-08-23 06:59:12,049 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/3e89c9e369cf4b3a92f291c8446086c0, entries=321984, sequenceid=99512270, filesize=8.6 M 2015-08-23 06:59:12,050 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134219432, currentsize=181.38 KB/185728 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 800ms, sequenceid=99512270, compaction requested=true 2015-08-23 06:59:12,053 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. 2015-08-23 06:59:12,053 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp, totalSize=127.1 M 2015-08-23 06:59:12,061 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 06:59:22,422 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. into bc97a4e88ce64caa912ed6a6ee3602f7(size=126.9 M), total size for store is 7.5 G. This selection was in queue for 0sec, and took 10sec to execute. 2015-08-23 06:59:22,422 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., storeName=e, fileCount=3, fileSize=127.1 M, priority=19, time=3850280464254545; duration=10sec 2015-08-23 07:00:47,093 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.08 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:00:47,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103589406, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/b342d844fe764fd2838ad16e97377e65 2015-08-23 07:00:47,958 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/b342d844fe764fd2838ad16e97377e65, entries=365284, sequenceid=103589406, filesize=8.7 M 2015-08-23 07:00:47,959 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134301760, currentsize=165.90 KB/169880 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 866ms, sequenceid=103589406, compaction requested=true 2015-08-23 07:01:50,371 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313031390 with entries=49323, filesize=121.69 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313310168 2015-08-23 07:02:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21810861, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39569, evicted=173, evictedPerRun=0.004372109659016132 2015-08-23 07:03:48,073 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:03:48,899 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101772530, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/1b4b89e4f99942cf81547a66f9c445a7 2015-08-23 07:03:48,951 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/1b4b89e4f99942cf81547a66f9c445a7, entries=386250, sequenceid=101772530, filesize=8.9 M 2015-08-23 07:03:48,952 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134217848, currentsize=86.06 KB/88128 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 879ms, sequenceid=101772530, compaction requested=true 2015-08-23T07:05:17.224+0000: [CMS-concurrent-mark-start] 2015-08-23T07:05:17.279+0000: [CMS-concurrent-mark: 0.056/0.056 secs] [Times: user=0.29 sys=0.00, real=0.05 secs] 2015-08-23T07:05:17.279+0000: [CMS-concurrent-preclean-start] 2015-08-23T07:05:17.283+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2015-08-23T07:05:17.283+0000: [CMS-concurrent-abortable-preclean-start] 2015-08-23T07:05:20.100+0000: [CMS-concurrent-abortable-preclean: 0.813/2.817 secs] [Times: user=1.06 sys=0.04, real=2.81 secs] 2015-08-23T07:05:20.130+0000: [CMS-concurrent-sweep-start] 2015-08-23T07:05:20.357+0000: [CMS-concurrent-sweep: 0.227/0.227 secs] [Times: user=0.25 sys=0.01, real=0.23 secs] 2015-08-23T07:05:20.357+0000: [CMS-concurrent-reset-start] 2015-08-23T07:05:20.378+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 07:06:31,649 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [172.24.32.16:10110, 172.24.32.10:10110, 172.24.32.7:10110] 2015-08-23 07:06:31,708 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313310168 with entries=49796, filesize=121.82 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313591393 2015-08-23 07:07:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21810861, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39599, evicted=173, evictedPerRun=0.004368797410279512 2015-08-23 07:11:11,419 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313591393 with entries=49689, filesize=121.67 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313871227 2015-08-23 07:11:16,549 INFO [sync.1] wal.FSHLog: Slow sync cost: 179 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.5:10110] 2015-08-23 07:11:16,549 INFO [sync.0] wal.FSHLog: Slow sync cost: 184 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.5:10110] 2015-08-23 07:11:16,549 INFO [sync.3] wal.FSHLog: Slow sync cost: 139 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.5:10110] 2015-08-23 07:11:16,549 INFO [sync.2] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.5:10110] 2015-08-23 07:11:16,549 INFO [sync.4] wal.FSHLog: Slow sync cost: 218 ms, current pipeline: [172.24.32.16:10110, 172.24.32.2:10110, 172.24.32.5:10110] 2015-08-23 07:11:43,854 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:11:44,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=99568426, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/.tmp/a6d761a9c4f14070a8b170ba303aa2d7 2015-08-23 07:11:44,675 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/6bd723149600afe2ecb4a9acc9fce2c7/e/a6d761a9c4f14070a8b170ba303aa2d7, entries=322569, sequenceid=99568426, filesize=8.6 M 2015-08-23 07:11:44,676 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134221592, currentsize=129.22 KB/132320 for region aeris,\x0E\x00\x00\x00\x00\x00\x00\x00\x0A/Sysdb/cell/1/agent/config/Pimsm,1439917707651.6bd723149600afe2ecb4a9acc9fce2c7. in 822ms, sequenceid=99568426, compaction requested=true 2015-08-23 07:12:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21810861, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39629, evicted=173, evictedPerRun=0.0043654898181557655 2015-08-23 07:12:53,325 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:12:54,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=92390066, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp/73b04229c88442359a320599e0c2bf2c 2015-08-23 07:12:54,183 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/e/73b04229c88442359a320599e0c2bf2c, entries=367575, sequenceid=92390066, filesize=8.7 M 2015-08-23 07:12:54,185 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218112, currentsize=108.84 KB/111456 for region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. in 860ms, sequenceid=92390066, compaction requested=true 2015-08-23 07:12:54,187 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. 2015-08-23 07:12:54,187 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/ff398ccd85feb23e75c0c4ce80e74cbe/.tmp, totalSize=26.1 M 2015-08-23 07:12:54,197 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 07:12:57,064 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe. into 5de64c5c4f264f159c7453ccffbbaf41(size=25.9 M), total size for store is 6.3 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-08-23 07:12:57,064 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x05\x00\x00\x00\x00\x00\x00\x00\x08/Sysdb/cell/1/agent/status/Coredump,1439174160675.ff398ccd85feb23e75c0c4ce80e74cbe., storeName=e, fileCount=3, fileSize=26.1 M, priority=20, time=3851102598650307; duration=2sec 2015-08-23 07:13:45,256 INFO [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:13:46,086 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=103657365, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp/4dc914b37eaa4b6e86f590c1c57c0b06 2015-08-23 07:13:46,138 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/e/4dc914b37eaa4b6e86f590c1c57c0b06, entries=365756, sequenceid=103657365, filesize=8.7 M 2015-08-23 07:13:46,139 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134236552, currentsize=129.49 KB/132600 for region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. in 883ms, sequenceid=103657365, compaction requested=true 2015-08-23 07:13:46,142 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. 2015-08-23 07:13:46,142 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4f9970eda1177fedb60b808a938a6c7d/.tmp, totalSize=60.4 M 2015-08-23 07:13:46,151 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 07:13:51,595 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d. into 3d45a95f108d49e58956ff90eaa2a09f(size=60.2 M), total size for store is 8.8 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-08-23 07:13:51,595 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x0F\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet20/intfCounter/current,1438329712419.4f9970eda1177fedb60b808a938a6c7d., storeName=e, fileCount=3, fileSize=60.4 M, priority=19, time=3851154553890949; duration=5sec 2015-08-23 07:15:51,875 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440313871227 with entries=49730, filesize=121.64 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314151725 2015-08-23 07:15:57,080 INFO [sync.1] wal.FSHLog: Slow sync cost: 236 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.10:10110] 2015-08-23 07:15:57,080 INFO [sync.0] wal.FSHLog: Slow sync cost: 250 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.10:10110] 2015-08-23 07:15:57,080 INFO [sync.2] wal.FSHLog: Slow sync cost: 215 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.10:10110] 2015-08-23 07:15:57,080 INFO [sync.4] wal.FSHLog: Slow sync cost: 256 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.10:10110] 2015-08-23 07:15:57,090 INFO [sync.3] wal.FSHLog: Slow sync cost: 220 ms, current pipeline: [172.24.32.16:10110, 172.24.32.9:10110, 172.24.32.10:10110] 2015-08-23 07:17:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21824307, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39659, evicted=173, evictedPerRun=0.004362187813967466 2015-08-23 07:17:51,653 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., current region memstore size 128.00 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:17:52,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=101845613, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp/fc95a78963a3456c88fba4072ffbdbfe 2015-08-23 07:17:52,439 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/e/fc95a78963a3456c88fba4072ffbdbfe, entries=386537, sequenceid=101845613, filesize=8.8 M 2015-08-23 07:17:52,440 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218144, currentsize=115.54 KB/118312 for region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. in 787ms, sequenceid=101845613, compaction requested=true 2015-08-23 07:17:52,442 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HRegion: Starting compaction on e in region aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. 2015-08-23 07:17:52,442 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Starting compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into tmpdir=hdfs://0.namenode.sjc.aristanetworks.com:8820/hbase/data/default/aeris/4691fd4d7548ec013bb9d0c89f18a5f3/.tmp, totalSize=44.0 M 2015-08-23 07:17:52,446 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=13, currentSize=5481968, freeSize=4284137744, maxSize=4289619712, heapSize=5481968, minSize=4075138560, minFactor=0.95, multiSize=2037569280, multiFactor=0.5, singleSize=1018784640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-08-23 07:17:56,288 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.HStore: Completed compaction of 3 file(s) in e of aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3. into b00390c1d10249408149bea9d09edfd7(size=43.8 M), total size for store is 8.0 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-08-23 07:17:56,288 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104-shortCompactions-1439917700316] regionserver.CompactSplitThread: Completed compaction: Request = regionName=aeris,\x08,1437814114755.4691fd4d7548ec013bb9d0c89f18a5f3., storeName=e, fileCount=3, fileSize=44.0 M, priority=19, time=3851400853660291; duration=3sec 2015-08-23T07:18:25.866+0000: [CMS-concurrent-mark-start] 2015-08-23T07:18:25.922+0000: [CMS-concurrent-mark: 0.055/0.055 secs] [Times: user=0.28 sys=0.00, real=0.06 secs] 2015-08-23T07:18:25.922+0000: [CMS-concurrent-preclean-start] 2015-08-23T07:18:25.925+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2015-08-23T07:18:25.960+0000: [CMS-concurrent-sweep-start] 2015-08-23T07:18:26.184+0000: [CMS-concurrent-sweep: 0.224/0.224 secs] [Times: user=0.24 sys=0.01, real=0.23 secs] 2015-08-23T07:18:26.184+0000: [CMS-concurrent-reset-start] 2015-08-23T07:18:26.205+0000: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2015-08-23 07:20:35,081 INFO [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314151725 with entries=50229, filesize=121.68 MB; new WAL /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 2015-08-23 07:21:30,088 WARN [ResponseProcessor for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 java.io.IOException: Bad response ERROR for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 from datanode 172.24.32.4:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828) 2015-08-23 07:21:30,089 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.4:10110: bad datanode 172.24.32.4:10110 2015-08-23 07:21:49,593 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.5:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:21:49,594 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.5:10110: bad datanode 172.24.32.5:10110 2015-08-23 07:21:52,642 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.1:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:21:52,642 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.1:10110: bad datanode 172.24.32.1:10110 2015-08-23 07:21:55,693 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.6:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:21:55,693 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.6:10110: bad datanode 172.24.32.6:10110 2015-08-23 07:21:58,718 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.8:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:21:58,720 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode 172.24.32.8:10110 2015-08-23 07:22:01,768 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.14:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:01,769 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.14:10110: bad datanode 172.24.32.14:10110 2015-08-23 07:22:04,806 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.2:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:04,806 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.2:10110: bad datanode 172.24.32.2:10110 2015-08-23 07:22:07,852 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.9:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:07,853 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.9:10110: bad datanode 172.24.32.9:10110 2015-08-23 07:22:10,877 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.3:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:10,878 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.3:10110: bad datanode 172.24.32.3:10110 2015-08-23 07:22:13,920 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.7:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:13,920 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.7:10110: bad datanode 172.24.32.7:10110 2015-08-23 07:22:16,945 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.10:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:16,945 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.10:10110: bad datanode 172.24.32.10:10110 2015-08-23 07:22:19,987 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.12:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:19,988 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.12:10110: bad datanode 172.24.32.12:10110 2015-08-23 07:22:23,012 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.11:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:23,012 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.11:10110: bad datanode 172.24.32.11:10110 2015-08-23 07:22:26,054 INFO [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 172.24.32.15:10110 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1203) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,055 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: Error Recovery for block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.15:10110: bad datanode 172.24.32.15:10110 2015-08-23 07:22:26,056 WARN [DataStreamer for file /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998 block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099] hdfs.DFSClient: DataStreamer Exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 WARN [sync.2] hdfs.DFSClient: Error while syncing java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 WARN [sync.3] hdfs.DFSClient: Error while syncing java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 WARN [sync.4] hdfs.DFSClient: Error while syncing java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 WARN [sync.1] hdfs.DFSClient: Error while syncing java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 WARN [sync.0] hdfs.DFSClient: Error while syncing java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,058 INFO [sync.3] wal.FSHLog: Slow sync cost: 70967 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110] 2015-08-23 07:22:26,058 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,057 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,058 INFO [sync.1] wal.FSHLog: Slow sync cost: 70908 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110] 2015-08-23 07:22:26,057 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,058 INFO [sync.0] wal.FSHLog: Slow sync cost: 70927 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110] 2015-08-23 07:22:26,058 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,058 INFO [sync.2] wal.FSHLog: Slow sync cost: 70987 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110] 2015-08-23 07:22:26,059 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,058 INFO [sync.4] wal.FSHLog: Slow sync cost: 70953 ms, current pipeline: [172.24.32.16:10110, 172.24.32.13:10110] 2015-08-23 07:22:26,058 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,059 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,059 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,060 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,060 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,060 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,060 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,060 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,061 WARN [B.defaultRpcServer.handler=49,queue=4,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475069,"responsesize":4867,"method":"Multi","processingtimems":70992,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,061 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,061 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,062 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,062 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,062 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,062 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,062 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,062 WARN [B.defaultRpcServer.handler=46,queue=1,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475209,"responsesize":4841,"method":"Multi","processingtimems":70853,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,063 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,063 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,063 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,063 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,063 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,063 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,064 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,064 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,064 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 WARN [B.defaultRpcServer.handler=27,queue=2,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475164,"responsesize":11616,"method":"Multi","processingtimems":70901,"client":"172.24.32.5:54577","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,065 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,065 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,066 WARN [B.defaultRpcServer.handler=31,queue=1,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475229,"responsesize":10646,"method":"Multi","processingtimems":70837,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,066 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,066 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,066 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,067 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,067 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,067 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,067 WARN [B.defaultRpcServer.handler=12,queue=2,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475189,"responsesize":10649,"method":"Multi","processingtimems":70878,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,067 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,067 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,068 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,067 WARN [B.defaultRpcServer.handler=10,queue=0,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475449,"responsesize":10649,"method":"Multi","processingtimems":70618,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,067 WARN [B.defaultRpcServer.handler=20,queue=0,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475424,"responsesize":8715,"method":"Multi","processingtimems":70643,"client":"172.24.32.5:54577","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,068 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,068 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,068 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,069 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,069 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,069 WARN [B.defaultRpcServer.handler=39,queue=4,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475309,"responsesize":29989,"method":"Multi","processingtimems":70760,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,069 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,070 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,070 WARN [B.defaultRpcServer.handler=14,queue=4,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475149,"responsesize":22256,"method":"Multi","processingtimems":70921,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,070 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,070 WARN [B.defaultRpcServer.handler=17,queue=2,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475090,"responsesize":35791,"method":"Multi","processingtimems":70980,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,070 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,070 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,071 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,071 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,071 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,071 WARN [B.defaultRpcServer.handler=28,queue=3,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475264,"responsesize":50297,"method":"Multi","processingtimems":70807,"client":"172.24.32.5:54577","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,073 WARN [B.defaultRpcServer.handler=35,queue=0,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475245,"responsesize":70605,"method":"Multi","processingtimems":70828,"client":"172.24.32.5:54577","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,073 WARN [B.defaultRpcServer.handler=26,queue=1,port=9104] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1440314475130,"responsesize":89946,"method":"Multi","processingtimems":70943,"client":"172.24.32.4:58196","queuetimems":0,"class":"HRegionServer"} 2015-08-23 07:22:26,085 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,085 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,087 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,087 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,089 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,089 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,091 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,092 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,093 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,093 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,093 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,093 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,095 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,095 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,097 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,097 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,101 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,101 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,102 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,102 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,106 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,106 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,108 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,108 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,110 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,111 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,112 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,112 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,113 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,113 ERROR [sync.3] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,114 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,114 ERROR [sync.4] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,124 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,125 ERROR [sync.0] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,125 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,125 ERROR [sync.1] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,125 FATAL [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1] wal.FSHLog: Could not append. Requesting close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:26,126 ERROR [sync.2] wal.FSHLog: Error syncing, request close of wal java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.24.32.16:10110, 172.24.32.13:10110], original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2015-08-23 07:22:40,419 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=5.23 MB, freeSize=3.99 GB, max=4.00 GB, blockCount=13, accesses=21830837, hits=345, hitRatio=0.00%, , cachingAccesses=490, cachingHits=289, cachingHitsRatio=58.98%, evictions=39689, evicted=173, evictedPerRun=0.00435889046639204 2015-08-23 07:24:40,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 4168 2015-08-23 07:24:45,043 INFO [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6., current region memstore size 52.92 MB, and 1/1 column families' memstores are being flushed. 2015-08-23 07:24:50,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 21382 2015-08-23 07:25:00,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 18144 2015-08-23 07:25:10,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 9856 2015-08-23 07:25:20,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 8583 2015-08-23 07:25:30,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 5880 2015-08-23 07:25:40,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 14781 2015-08-23 07:25:50,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 22865 2015-08-23 07:26:00,879 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 12097 2015-08-23 07:26:10,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 22751 2015-08-23 07:26:20,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 17838 2015-08-23 07:26:30,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 21306 2015-08-23 07:26:40,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 8307 2015-08-23 07:26:50,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 14121 2015-08-23 07:27:00,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 14100 2015-08-23 07:27:10,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 21293 2015-08-23 07:27:20,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 3828 2015-08-23 07:27:30,875 INFO [r12s16.sjc.aristanetworks.com,9104,1439917659071_ChoreService_1] regionserver.HRegionServer: r12s16.sjc.aristanetworks.com,9104,1439917659071-MemstoreFlusherChore requesting flush for region aeris,\x10\x00\x00\x00\x00\x00\x00\x00\x01/Sysdb/interface/counter/eth/slice/phy/1/intfCounterDir/Ethernet34/intfCounter/lastClear,1439083371056.a1233a97ab54c0f50ac9a39cd05f75a6. after a delay of 6572